Integrating Active Tangible Devices with a Synthetic Environment for

Collaborative Engineering

Sandy Ressler Brian Antonishek Qiming Wang Afzal Godil

National Institute of Standards and Technology

Jared FreelandDAS FA

CIS 4930

Abstract of the AbstractAbstract of the Abstract

This paper describes the creation of an environment for collaborative engineering in which the goal is to improve the user interface by using haptic manipulation with synthetic environments.The system to be outlined combines some of what Dr. Fishwick discussed on Wednesday, as well as Scott’s presentation on “Real Reality”.

IntroductionIntroduction

The immediate goal: to determine the feasibility The immediate goal: to determine the feasibility of using a tangible interface with a multiuser of using a tangible interface with a multiuser VRML environment as applied to collaborative VRML environment as applied to collaborative engineering.engineering.

By “tangible” we refer to the ability to pick up and By “tangible” we refer to the ability to pick up and interact with actual physical objects represented interact with actual physical objects represented in the virtual environmentin the virtual environment

IntroductionIntroduction

A secondary goal of the project was to use as A secondary goal of the project was to use as much off-the-shelf software and hardware as much off-the-shelf software and hardware as possible, to facilitate transfer of the technology possible, to facilitate transfer of the technology into the commercial world.into the commercial world.

The mediation hub is run by JavaThe mediation hub is run by Java

The VE uses a commercial system, the blaxxun The VE uses a commercial system, the blaxxun Community PlatformCommunity Platform

The tangible devices are off-the-shelf The tangible devices are off-the-shelf configurable robots by LEGO Mindstormsconfigurable robots by LEGO Mindstorms

System OverviewSystem Overview

The overall environment is conceptually simple.The overall environment is conceptually simple.

Two collaborating engineers in geographically Two collaborating engineers in geographically separate areas wish to manipulate and discuss a separate areas wish to manipulate and discuss a construction projectconstruction project

Recent work at NIST has demonstrated that Recent work at NIST has demonstrated that VRML can be used to represent rich VRML can be used to represent rich construction environments, but manipulation of construction environments, but manipulation of elements such as a virtual excavator is elements such as a virtual excavator is awkward.awkward.

System OverviewSystem Overview

Control panels with many buttons and sliders are Control panels with many buttons and sliders are functional but can be difficult to manipulate.functional but can be difficult to manipulate.

The answer: direct haptic manipulation should The answer: direct haptic manipulation should be more intuitive for interaction.be more intuitive for interaction.

Users move the tangible excavator and adjust Users move the tangible excavator and adjust the rotatable arm, causing the virtual “mirror” to the rotatable arm, causing the virtual “mirror” to update.update.

System OverviewSystem Overview

The core of the system is the The core of the system is the Java based Virtual Environment Device IntegrationJava based Virtual Environment Device Integrationserver, or JVEDI, which acts as a hub between all server, or JVEDI, which acts as a hub between all the system’s components. the system’s components.

The server runs as a stand-alone Java application The server runs as a stand-alone Java application on the host computer. on the host computer.

System OverviewSystem Overview The Real EnvironmentThe Real Environment

Two work surfaces (A.K.A. tables)Two work surfaces (A.K.A. tables)

A LEGO Mindstorm robot on each surfaceA LEGO Mindstorm robot on each surface

Above each surface is a video camera looking Above each surface is a video camera looking down at the surface, providing 2D down at the surface, providing 2D position/orientationposition/orientation

System OverviewSystem Overview The Virtual EnvironmentThe Virtual Environment

A multi-user blaxxun environment displays the A multi-user blaxxun environment displays the sum of both (or all) physical environmentssum of both (or all) physical environments

A simplified user interface consisting of buttons A simplified user interface consisting of buttons and arrows is included for collaborators without and arrows is included for collaborators without access to an actual robotaccess to an actual robot

Interesting Points of the SystemInteresting Points of the System

Unlike “graspable” interfaces, this system does Unlike “graspable” interfaces, this system does not use a haptic glove or data glove of any kind.not use a haptic glove or data glove of any kind.

Instead, by using a camera to track the Instead, by using a camera to track the movement of robots, the user is given complete movement of robots, the user is given complete unrestricted control of the robotsunrestricted control of the robots

Interesting Points of the SystemInteresting Points of the System

The virtual and real environments are kept The virtual and real environments are kept synchronized. They always mirror each other.synchronized. They always mirror each other.

This is accomplished by always using position This is accomplished by always using position values reported by the video system.values reported by the video system.

Integration IssuesIntegration Issues

The most challenging aspect in creating the The most challenging aspect in creating the work environment was integrating all the work environment was integrating all the processing elements.processing elements.

Functionality for controlling robots and reading Functionality for controlling robots and reading and writing data from a position tracking device and writing data from a position tracking device had to be built for VRML’s External Authoring had to be built for VRML’s External Authoring Interface.Interface.

A fully configured version of the environment A fully configured version of the environment requires up to six separate computers.requires up to six separate computers.

Major ComponentsMajor Components Vision ProcessingVision Processing

The position and orientation of the LEGO robots The position and orientation of the LEGO robots are computed in real time using a computer are computed in real time using a computer vision method based on color tracking.vision method based on color tracking.

The vision program uses an inexpensive camera The vision program uses an inexpensive camera and can track multiple robots at 10 frames/secand can track multiple robots at 10 frames/sec

Major ComponentsMajor Components Vision ProcessingVision Processing

To track the LEGO robots, two differently To track the LEGO robots, two differently colored cards were attached to the robot. colored cards were attached to the robot.

The computer vision program uses probability The computer vision program uses probability distribution to find the centers of the two distribution to find the centers of the two squares, the mean of which is the robot’s squares, the mean of which is the robot’s position.position.

The orientation is the arctangent of the The orientation is the arctangent of the difference of the centersdifference of the centers

Major ComponentsMajor Components Speech InputSpeech Input

A user who is moving robots cannot easily A user who is moving robots cannot easily access a keyboard.access a keyboard.It can also be necessary to move robots on two It can also be necessary to move robots on two surfaces simultaneously.surfaces simultaneously.Voice commands were built in such as “forward”, Voice commands were built in such as “forward”, “backward”, “left”, “right”, “select red”, “select “backward”, “left”, “right”, “select red”, “select blue”, etc.blue”, etc.

Major ComponentsMajor Components Multiuser VRMLMultiuser VRML

The multiuser aspect of the VRML world was The multiuser aspect of the VRML world was accomplished with commercially available accomplished with commercially available software from blaxxun Community.software from blaxxun Community.The first part is the virtual world which includes The first part is the virtual world which includes the two lego robots (red and blue), the floor, and the two lego robots (red and blue), the floor, and a cylinder.a cylinder.The second part is the control panelThe second part is the control panel

Auxiliary ProcessingAuxiliary Processing

Collision DetectionCollision Detection

Suppose the robots on two separate work Suppose the robots on two separate work surfaces collide virtuallysurfaces collide virtually

The collision detection is performed by the The collision detection is performed by the VRML world, and knowledge of it exists only in VRML world, and knowledge of it exists only in the VE.the VE.

The robots light up and beep when they hit The robots light up and beep when they hit something in the virtual world.something in the virtual world.

Auxiliary ProcessingAuxiliary Processing

Task Commands and RecordingTask Commands and Recording

Small programmatic tasks were created for the Small programmatic tasks were created for the robots, such as movement patterns.robots, such as movement patterns.

Additionally, functionality was added that allows Additionally, functionality was added that allows users to record the movements of the robots, to users to record the movements of the robots, to be played back and repeated later.be played back and repeated later.

LinksLinks

The JVEDI code is publicly available atThe JVEDI code is publicly available athttp://ovrt.nist.gov/jvedihttp://ovrt.nist.gov/jvedi

More on robots at legomindstorms.comMore on robots at legomindstorms.com

More on the blaxxun Community at blaxxun.comMore on the blaxxun Community at blaxxun.com

DiscussionDiscussion

How does this project relate or compare to How does this project relate or compare to what Scott discussed Wednesday?what Scott discussed Wednesday?

What are the advantages/disadvantages?What are the advantages/disadvantages?

What can and needs to be improved?What can and needs to be improved?