tools for edutainment sensory interfaces, augmented reality and edutainment

of 100 /100
Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment

Author: kristopher-ethelbert-jordan

Post on 11-Jan-2016




7 download

Embed Size (px)


  • Sensory Interfaces, Augmented Reality and Edutainment

    Tools for Edutainment

  • BackgroundWu Pin Fong

    Virtual Reality Consulting company, Xtensory Inc, California, America

    Total 25 years industrial experienceDone virtual reality a long time 14 years

    Emphasis mixed reality support for industry

  • Virtual Reality Background

  • StartedWorked at Digital Equipment (now part of Hewlett Packard) 1990

    Start the virtual reality program at Digital

    First major US company to support virtual reality

  • Undocumented VR HistoryWorking for a major american computer company

    Able to work with all the key VR developers

    VR small community then

  • RelationshipsJaron Lanier, VPL ResearchScott Fisher, NASADr Tom Furness, HITL Labs, University of WashingtonCarolina Cruz , CAVE inventor, Iowa State UniversityRudy Darken, Naval Post Graduate School

  • OthersWilliam BrickenAutodeskHITL Labs

    Foundation for WorldToolKit

    Device manufacturers Ascension, Polhemius, Intersense, Immersion, etc

  • Jaron Lanier VPL ResearchTaught me Virtual Reality

    Worked on VR project For SIGGRAPH 1990

  • VPL Research in 1990Very exciting

    New technology

    Mark Bolas (later to start FakeSpace)

  • Tom Furness Director, HITL LabsCame from Aerospace Medical Research Laboratory (USAF), Wright-Patterson AFB, OH.Helped get funding from Digital Equipemt for HITL Labs

  • Electronic Visualization Lab - CAVEThe CAVE is a projection based virtual reality system developed at the Electronic Visualization Lab created by Carolina Cruz-Neira

  • Naval Post Graduate SchoolRudy Darken, Director MOVES Institute, Monterey Post Graduate School

    Edutainment game - America's Army

  • Some ProjectsNASAAmerican army, navyWright patterson air force base, OHFAABank of AmericaMayo ClinicToshibaHITL labs, University of WashingtonHasbroLockheed SchlumbergerMonterey naval schoolHeitmanCompaq

  • OverViewNeed make it easy to develop mixed reality games

    Provide Standard components

  • Why components Many mixed reality games involve people who are not virtual reality experts

    Most edutainment projects assume completely new softwareControl of actual real dataExpert programmers

    Real world need interface with legacy software

  • Edutainment games

    Out of research into actual industrial use

  • TodayDiscuss one standard component - device supportExample of its useSpecial device support HapticsA haptics exampleComponent for building virtual cities for use in mixed reality games

  • Device Support Component

  • XvsLink Standard device component for device support

    Paper - A Component-based Sensor Architecture for CAVEs (CAVE Automated Virtual Environments), Proceedings of Immersive Projection Technology Workshop, Iowa State University, 2000.

  • Industry tested since 1996Not research Used in actual products

    ToshibaUniversity of Pennsylvania virtual human JackMayo Clinic

  • Standard Device InterfaceImportance of components

    Example of components standardised device interfaces

  • XvsLink PurposeInstrument existing legacy software

    Linux, Windows, Sun Solaris, HP, SGI

    Easy to use VR devices

    Interfaces to non VR devices

  • Work Closely with Device ManufacturersSo closely actually referenced in their manuals Example Intersense IS600 manual

  • Focused - Does not provideuser interaction techniques

    scene graph (or transform tree)

    graphics techniques

  • Provideshandle the device control layer

    provide communication between hosts

    hook to higher levels of an edutainment system

  • XvsLink Functional DiagramLegacy softwareDevice supportSession capture &playbackNetwork supportUsersDevelopersadministratorsVirtual vehicleUser interaction analysis

  • One industrial DifferenceNeed to monitor user interaction with the game

    Capture user data for later playback and analysis

  • Layered ArchitectureXvsLinkApplicationdeviceinterfacetransportUser interaction toolsLegacy software

  • Component Device InterfaceProvide Easy Access Common Interface to different devices Access to VR peripherals from any application

    Storage and replay of device data

  • XvsLinkXvsLink is a C++ class library with both a C and C++ interface that provides a standard device interface for integrating devices into simulator applications.

  • Make devices invisible to userWriting a driver for a new device usually a few days Special commands for specific device drivers

    Windows and linux kernel level drivers

  • OperationOpen networked, serial, USB, shared memory connections

    Invisible to user Generic devices by functions

    connection automatically restored if the device server is shut down and restarted

  • Design

    generic devicespolled or queuedlocal or remoteactual or virtualdevices identified by logical name

  • Design

    polled or queued- application can specifypolledreads data asynchronouslytypical example: tracker, valuatorqueuedcirculartypical example: keyboard, button

  • Local and Remote devicesremote devices treated as local devices Local client and server queries for device capabilities

  • Design

    local or remote deviceapplication unaware of source

    can use actual or virtual dataDevice data from hardware or calculations

  • Device Mapping

    interfaces specify a set of capabilitiesdevice is mapped onto one or more interfacesClient connects to the different interfacesrun-time bindings

  • Design

    identified by logical name

    Device can map to any name

    new names can be created as needed

  • Extensible

    Generic Type devices

    Layered devices Multiple-behavior devices

  • Design

    generic deviceslocatorvaluatorbutton

  • Device added dynamicallydynamic device support

    examplesIntersense trackerFakespace wand, pinchGlovePhantom haptic

  • Support

    Linux, windows, SUN, HP, SGI

    hooks to other tools, packages

    Hooks for legacy systems

  • Example of XvsLinks Use

  • Virtual Collaborative Clinic

  • NASA Virtual Collaborative Clinic1998

    Dr. Muriel Ross, head of NASA's biometrics research at Ames

    Out of research into deployment usually no news

  • Virtual Collaborative clinic

  • Collaboration Partners

  • ObjectivesMedical support in remote locations

    surgical try-outs in mixed reality after which the most adequate procedure can be stored to be used in the actual operation.

    project computer images onto the patient in order to guide physicians during surgical interventions.

  • Special device support Haptics

  • Haptic Rendering of Virtual Stimuli

  • Haptic

  • Haptic InformationHaptic information is a form of sensory stimulation that is acquired through collisions with objects

  • Using the Haptic Sensory Channel

    Force, vibration, textureobject shape, compliance, impact, contact, sliding, slipping, torque and kinematic constraint. encoding the feel and movement of real or virtual objects during manipulation or exploration

  • Haptic Exploratory Procedures

  • Human Sensing

  • Force Feedback

  • Force Feedback

  • Rutgers

  • Haptics

  • Example

  • Xtensorys Tactile Controller

  • Shape Memory Alloy (SMA)a system of long stroke vertical pin-rods, using a coil-form Shape Memory Alloy (SMA) as an actuator.

    This material can be stretched or deformed from its original shape but would spontaneously return to its original shape when heated.

  • Tactor Sensor

  • Tactor Arraya 16 by 16 array of pin-rods

    high-speed update rates

  • Tactor Array in Action

  • Operationorchestrating the vertical motion of a dense array of pin-rods. The effect is similar to a children's "Pop up book", where a flat surface transforms into a 3D form. the driving method is scalable so that tactor arrays of any size can be constructed

  • A Haptics Example

  • Call Options Game

    Done for major american bank to train its traders

    Conceptual architecture and framework for mixing real and virtual data

    Added haptically rendering virtual stimuli

  • The Game - Call OptionsThe options portfolio consist of options with 3 different times to expiration. The gamma value of an option is increasing dramatically just before expiration if the option is about at-the-money. Local peaks or valleys therefore occur for each strike, near to maturity . (Long positions give positive gamma and short positions give negative gamma).

  • Using HapticsPeaks and valleys shown visually rendered hapticallyMix real and virtual data

  • In Operation

  • Another Haptics Example

  • Enclosed Hazardous Spaces

  • Narrow field of View

  • Constrained SpaceNeed sense of touch to movehaptic orienting cues play a critical role

  • Inspection GameMix real object containment vessel

    Virtual hazards

    Tactile sensing


  • Virtual City Support Component

  • WhyMany mixed reality games will use actual city buildings, streets, locations

    Need make it easy to mix real and virtual for actual cities

    Military urban operations

  • Urban OperationsIn the last 10 years, however, the U.S. has committed military forces to far-flung urban operations in Panama, Somalia, Liberia, Bosnia and now Kosovo. The Marine Corps alone has been involved in urban operations in 237 of its last 250 deployments.

  • Urban OperationsChracterised by extremely short firing ranges imposed by a city's long canyons of high-rise buildings and dense thickets of trackless housing and teeming alleys

    Need for local infantry autonomy

    No command and control capability

  • Intelligence & Reconnaissance

  • RequirementsQuick and easy way to build virtual cities for training games

    Acuisition of motion imagery from camera phones, UAVs

    Build virtual cities from analysis of motion imagery

    Time Frame - Late 1990s

  • Xtensory ContractsEnhanced Multiplatform, Integrated Virtual Urban Warfare Simulation. Navy SBIR N99-063.

    Soldier-Computer Interface for Enhanced Interaction and Communication. Army SBIR A95-035

    Immersive Visualization of Complex Situations for Mission Rehearsal. Army SBIR A95-089.

  • ComponentMotion Imagery analysis (acquired from another company)

    Polygon processing into low polygon count models, autocad, 3DS file formats

    Good way to build virtual cities

  • Conclusion

    Shown how use of component tools help in making industrial strength edutainment games