iua awk11 virtualreality - david gameziua.davidgamez.eu/documents/iua_awk11_virtualreality.pdf ·...

1

David Gamez IUA Week 11 Autumn 2007 1

Notes

• Deadline for essay is 5pm today.

• Deadline for website assignment is next Thursday, 6th December, when there will be a short oral assessment.

• The rest of the class on the 6th December will be dedicated to explaining the exam and going over the material from the last 11 weeks.


Virtual Reality


Overview

• Allows a user to interact with a computer-simulation of a real or imagined environment.

• Buzzword in the 90’s, now used with more caution.

• Technology is real and there are an increasing number of applications.


Overview

• Several related technologies:– Virtual reality

– Telepresence

– Tele-immersion

– Augmented reality

• All depend on similar technology, so will cover this first.

• Will then explain each area individually.

• Second half of the lecture will give some applications.


Technology


Technology

• Virtual reality and its variants immerse the user in a computer-generated environment, which may be combined with the real world.

• To be realistic, the environment must track the user’s movements and update the display accordingly.

2


Technology

• For example, if the virtual environment consists of a single pedestal in the middle of a room, a user should be able to view the pedestal from any angle and the point of view should shift according to where the user is looking.

• If the environment incorporates 3-D sound, the user must be convinced that the sound’s orientation shifts in a natural way as he manoeuvres through the environment.


Sensory Input

• Stereoscopic displays.

• Audio.

• Tactile information.

• Force feedback.


Head Mounted Display

• Key part of most virtual reality setups.

• Headset with two monitors.

• Each image is slightly offset to give the illusion of depth.

• Usually LCD monitors.


Head Mounted Display


Retinal Scanning Displays

• Can also use a laser to project the image on the back of the eye.

• The image is drawn directly on to the retina by a low-powered laser scanning backwards and forwards. Because the laser moves so rapidly, the human visual system sees a complete and stable image.

• Has a wide field of view - 120 degrees as opposed to 25 degrees of the typical screen based display.


Retinal Scanning Display

3


Nomad Retina Scanning Display


Cave Virtual Environments

• Cave displays give users a much wider field of view, which helps with immersion.

• Also allows a group of people to share the experience at the same time.

• Stereoscopic glasses are used to give illusion of depth.


Virtual Reality Cave


Force Feedback

• High speed vibrations mimic the experience of moving hand over a rough surface.

• Steering wheel and other controls have been developed with force feedback.

• Boom arms can give force feedback.


Interaction with Environment

• Need to track the movements of the user in order to present different aspects of the environment.

• Tracking systems analyze the orientation of a user’s point of view so that the computer system sends the right images to the visual display.

• Variety of different technologies are being developed.


Interaction with Environment

• Joystick

• Wired glove

• Boom arm

• Treadmill

• Controller wands

• Data suit

4


Wired Glove

• Glove-like input device for virtual reality environments.

• Various sensor technologies used to capture physical data such as bending of fingers.

• Often a motion tracker, such as a magnetic tracking device or inertial tracking device, is attached to capture the global position / rotation of the glove.

• Can also provide tactile feedback.


Wired Glove


Wii Wand Controller

• Tracking device included with the Nintendo Wii.

• As you turn and move the wand, your tennis racket, sword etc. turn and move on screen.


Wii Controller


Datasuit

• Whole body suit that captures the movements of the users body.

• Similar to wired glove.


Data suit

5


Biosensors

• Detect and interpret nerve and muscle activity.

• May be attached directly to the skin of a user, or may be incorporated into gloves or bodysuits.

• One limitation to biosensor suits is that they must be custom made for each user or the sensors will not line up properly on the user’s body.

• At an early stage of development.


Boom Arm

• Mechanical method of tracking the user’s movements.

• Precise and can also be used for force feedback.


Boom Arm


Treadmill

• Allows user to walk or run through the virtual environment.

• Ideally should be omni-directional so the user can walk or run in any direction.

• Exercise treadmills are now being fitted with virtual reality so that users can jog through a jungle or across an alien planet.


Treadmills


Computer Technology

• Lots of processing required to generate the virtual environment – especially graphics.

• With telepresence and tele-immersion, there are also high demands on the network.

• With augmented reality, the computer often has to be carried around by the user, which constrains its weight, power requirements and processing power.

6


Technology Issues: Latency

• Lag time between when a user acts and when the virtual environment reflects that action is called latency.

• Usually refers to the delay between the time a user turns his head or moves his eyes and the change in the point of view, though the term can also be used for a lag in other sensory outputs.

• Studies with flight simulators show that humans can detect a latency of more than 50 milliseconds.

• When a user detects latency, it causes him to become aware of being in an artificial environment and destroys the sense of immersion.

• Latency can also induce a feeling of nausea in the user.


Technology Issues: Tracking Accuracy

• The biggest challenge facing developers of augmented reality is the need to know where the user is located in reference to his or her surroundings.

• There's also the additional problem of tracking the movement of users' eyes and heads.

• Currently, both video see-through and optical see-through displays typically have lag in the overlaid material due to the tracking technologies currently available.


Technology Issues: Tracking Accuracy

• Tracking of the user’s movements can be done accurately in a constrained environment.

• Outside it is much more difficult.

• GPS is not accurate enough on its own

• The military uses multiple GPS signals.

• There is also differential GPS, which uses an area that has already been surveyed.


Virtual Reality on the Web

• Several different computer languages and web browsers have been developed to enable virtual environments to be viewed online.

• These include:– Virtual Reality Modeling Language (VRML)- the

earliest three-dimensional modeling language for the Web.

– 3DML - a three-dimensional modeling language where a user can visit a spot (or Web site) through most Internet browsers after installing a plug-in.

– X3D - the language that replaced VRML as the standard for creating virtual environments in the Internet.


Virtual Reality on the Web

• Other companies, such as Linden Labs, provide specialised software that enables users to interact within a single online virtual world.


Virtual Reality

7


Virtual Reality

• A computer generated immersive environment.

• Simulated, three-dimensional world that a user can manipulate and explore whilst feeling as if he were in that world

• Three-dimensional images appear to be life-size from the perspective of the user.

• Environment is entirely computer generated and does not correspond in real time to any actual environment.


Virtual Reality

• Also called cyberspace (a word invented by science fiction author William Gibson).

• In a virtual reality environment, a user experiences immersion, or the feeling of being inside and a part of that world.

• He is also able to interact with his environment in meaningful ways.

• The users’ motions are tracked, particularly their head and eye movements, and the images on the user's display are adjusted to reflect the change in perspective.


Virtual Reality


Telepresence


Telepresence

• Virtual reality technology is used to give the user the illusion that they are in a different location.

• Cameras capture data from the real location and the images are displayed in the user’s head mounted display.

• Force feedback may be used as well.

• Robot technology is used to carry out actions at the remote location.


Telepresence

• For example, imagine the user is controlling a humanoid robot.

• A datasuit records movements of the user’s body, which are used to control the movement of the robot.

• Images from cameras in the robot’s eyes are sent to the user.

• When the robot encounters resistance in its environment, force feedback is given to the user.

8


Telepresence

• The user sees and feels the remote environment as if he or she were actually there.

• The user can also interact with the environment.

• The real environment can be much bigger or smaller than the user.


Telepresence

• Very useful in a hazardous environments:

– Bomb disposal

– Handling radioactive waste

– Fire fighting

– Combat

• Very useful when the real environment is much bigger or smaller than the user, for example, delicate surgery.


Tele-immersion


Tele-immersion

• Enables users at geographically distributed sites to collaborate in real time in a shared, simulated environment as if they were in the same physical room.

• Intended for teleconferencing.

• People feel as if they are together in the same shared office space.


Tele-immersion

• Creates a sense of shared presence with distant individuals and their environments that feels substantially as if they were in one's own local space.

• Differs significantly from conventional video teleconferencing in that the user's view of the remote environment changes dynamically as the user moves her head.

• Blending of real and synthetic data.


Tele-immersion

9


Tele-immersion


Tele-immersion

• Blurs the lines between real and computer-generated images.

• Instead of commuting, people could attend board meetings by projecting themselves into the company's conference room.

• If your job requires you to travel, you could still be home for dinner by tele-immersing yourself into the family kitchen.


AugmentedReality


Augmented Reality

• Superimposes graphics, audio and other sense enhancements over a real-world environment in real-time.

• Graphics are changed to accommodate a user's head- and eye- movements, so that they are always appropriate.


Augmented Reality

• Gives people instant information that is relevant to their context and requires no research on their part.

• Typically uses:– Head-mounted display

– Tracking system

– Mobile computing power

• Should instantly recognize what someone is looking at, and retrieve and display the data related to that view.


Augmented Reality

10


Applications


Games


Games and Virtual Reality

• Virtual reality produces more realistic and immersive games and the technology has often been driven forward by game manufacturers.

• For example, force feedback devices in game controllers and the Nintendo Wiiwand.


Augmented Reality and Gaming

• Augmented reality enables video games to be taken outside.

• Monsters etc. can be superimposed on the environment.

• One Australian researcher created a prototype game that combines Quake with augmented reality.

• He put a model of a university campus into the game's software. Now, when he uses this system, the game surrounds him as he walks across campus.

• Also Human PacMan.


Augmented Reality Quake


Human PacMan

11


Medicine


Medical Training

• Students cannot be trained on real subjects.

• Corpses are expensive and unrealistic.

• Traditional videotaped instruction has a number of limitations:

– Fixed point of view

– Lack of depth perception and interactivity

– Missing information


Medical Tranining

• Virtual environments can be used to train everything from surgical procedures to diagnosing a patient.

• With virtual reality, it is possible to witness and explore (in time and space) a past surgical procedure, with the added benefit of instruction from the original surgeon or another instructor, as well as integrated 3D illustrations, annotations, and relevant medical metadata.


Medical Training

• Trainees should be able to freely and naturally walk around a life-sized, high-fidelity, 3D graphical reconstruction of the original events, pausing or stepping forward and backward in time to satisfy curiosity or allay confusion.

• A virtual-reality training system has been developed to teach surgical trainees how to set up an operating room properly for different types of surgery—a real benefit, as operating theatres cost around $30 a minute to run.


Medical Training

• Many VR training systems include devices such as scissors and drills which provide force-feedback.

• Trainees can feel what it is like to puncture an abdomen with a needle, collect bone marrow for transplant (difficult for the surgeon and often painful for the patient) or even drill into healthy and diseased parts of a tooth.

• Trainees can practice as many times as they like, and get feedback and precise measurements of their skill.


Robotic Surgery

• Surgeons are starting to use virtual reality technology to perform surgery remotely using robotic devices.

• Robots under control of voice and hand signals.

• Example of tele-presence

• Allows for high level of control and precision of surgical instruments in minimally invasive procedures.

• Has been used to position an endoscope, perform gallbladder surgery and correct gastroesophogeal reflux and heartburn.

12


Advantages of Robotic Surgery

• Allows the human surgeon to get closer to the surgical site than human vision will allow, and work at a smaller scale than conventional surgery permits.

• For example, in finger reattachment in children, the sutures can be a third to two-thirds of a millimetre in size.

• Fewer people needed in operating room.



• Can compensate for tremors in the doctor’s hands.

• Magnifies the force feedback surgeons receive, so they can get a better feel for the fragile and minute structures they are working on.

• Can be performed remotely (with potential latency problems), so a specialist can carry out the procedure across the country.



• At the University of Tokyo, researchers are working on a system that creates a stabilised image of a beating heart and a tool that translates surgical manoeuvres to compensate for the movement of the heartbeat.

• Can integrate medical imagery with the camera images (augmented reality) to increase the surgeon’s knowledge of the patient’s medical history and body.

• Allows the procedure to be recorded from the surgeon’s point of view, which is useful for training students.


Augmented Reality in Medicine


Da Vinci Surgical System

• Robotic surgery system.

• Surgeon manipulates instruments using joystick controls.

• His commands are used to control the robotic instruments inside the patient.



13


Surgeon Console & Patient Side Cart

• Surgeon Console– Surgeon operates while seated comfortably at a console viewing a 3-D

image of the surgical field.– Surgeon's fingers grasp the master controls below the display, with

hands and wrists naturally positioned relative to his or her eyes.– The system seamlessly translates the surgeon's hand, wrist and finger

movements into precise, real-time movements of surgical instruments inside the patient.

• Patient-side Cart– Provides either three or four robotic arms—two or three instrument

arms and one endoscope arm—that execute the surgeon's commands.– The laparoscopic arms pivot at the 1-2 cm operating ports, eliminating

the use of the patient's body wall for leverage and minimizing tissue damage.

– Supporting surgical team members assist in installing the properinstruments, prepare the 1-2 cm port in the patient and supervise the laparoscopic arms and tools being utilized.




Instruments & Vision System

• Instruments– A full range of instruments is available to support the surgeon

while operating.– Instruments are designed with seven degrees of motion that

mimic the dexterity of the human hand and wrist.– Each instrument has a specific surgical mission such as

clamping, suturing and tissue manipulation.– Quick-release levers speed instrument changes during surgical

procedures.

• Vision System– The Vision System, with high-resolution 3-D endoscope and

image processing equipment, provides the true-to-life 3-D images of the operative field.

– Operating images are enhanced, refined and optimized using image synchronizers, high-intensity illuminators and camera control units.




Tele-surgery

• Systems like Da Vinci also enable a doctor to perform delicate surgery miles away from the patient.

• If it were possible to use the computer console to move the robotic arms in real-time, then it would be possible for a doctor in California to operate on a patient in New York.

• A major obstacle in tele-surgery has been the time delay between the doctor moving his or her hands to the robotic arms responding to those movements.

• Currently, the doctor must be in the room with the patient for robotic systems to react instantly to the doctor's hand movements.


Augmented Vision for Visually Impaired

• Augmented reality can be used to improve vision for the visually impaired

• Can enhance the outline of a scene using a see-through, head-mounted display.

• Can also correct tunnel vision.

14


Augmented Vision for Visually Impaired


Treatment of Phobias

• VR technology can also be used to treat people with phobias and other psychological conditions.

• Virtual environments are used to expose the patient progressively to stimuli that cause him distress.

• Much more convenient than real exposure therapy.

• Patients are more willing to try the therapy because they know it isn't the real world.


Virtually Better

• Company that creates virtual reality environments for use in the treatment of anxiety disorders such as fear of public speaking, fear of flying, fear of heights, as well as post-traumatic stress disorder.

• www.virtuallybetter.com


Flight Simulators


Flight Simulator

• Flight simulators are a good example of a VR system that is effective within strict limits.

• In a good flight simulator, a user can take the same flight path under a wide range of conditions.

• Users can feel what it's like to fly through storms, thick fog or calm winds.

• Realistic flight simulators are effective and safe training tools, and though a sophisticated simulator can cost tens of thousands of dollars, they're cheaper than an actual aircraft.

• Very effective commercial use of the technology


Flight Simulator

15


Architecture & Design


Virtual Architecture

• Create virtual models of a building plans so that people can walk through the structure before the foundation is laid.

• Clients can move around exteriors and interiors and ask questions, or even suggest alterations to the design.

• Virtual models can give you a much more accurate idea of how moving through a building will feel than a miniature model.


Virtual Architecture


Augmented Reality and Architecture

• Augmented reality is useful at the construction and maintenance phases of a project.

• Markers can be attached to a particular object that a person is working on, and the augmented-reality system can draw graphics on top of it.

• Virtual model can be overlaid over the foundations as the building is constructed.


Virtual Cars

• Car companies have used VR technology to build virtual prototypes of new vehicles, testing them thoroughly before producing a single physical part.

• Designers can make alterations without having to scrap the entire model, as they often would with physical ones.

• The development process becomes more efficient and less expensive as a result.


Domestic

16


Domestic Telepresence

• Can be used to remotely be present in the home for security, to communicate with family etc.

• iRobot has developed a new telepresencerobot for this purpose.

• Allows you to navigate the home, find family members and engage in conversation.


iRobot Telepresence Robot


Military


Training

• The military have long been supporters of VR technology and development.

• Training programs can include everything from vehicle simulations to squad combat.

• VR systems are much safer and, in the long run, less expensive than alternative training methods.

• Soldiers who have gone through extensive VR training have proven to be as effective as those who trained under traditional conditions.


VR Parachute Trainer


Recruitment

• Virtual reality is also being used to recruit troops.

• America’s Army is an online multiplayer game used in recruitment.

• 5.4 million registered users.

• See: www.americasarmy.com

17


America’s Army


Augmented Reality

• Used in military vehicles and planes to provide information to the driver or pilot.

• Saves them looking at dials etc.

• Also used to identify targets etc. in a fast moving and confusing combat environment.


Nomad Retina Scanning Display


Telepresence

• Much safer for soldiers to be away from the scene of combat.

• Use robotic vehicles to provide telepresence.

• For example, unmanned aerial vehicles and the Foster-miller Talon.


Unmanned Aerial Vehicles (UAV)

• Military planes under remote control.

• Control room is similar to a flight simulator.


Predator UAV

18


UAV Control Station


Foster-Miller TALON

• Weapons platform operated by remote control.

• Soldiers use control station to manoeuvre robot, fire weapons, etc.


Foster-Miller TALON


Windowless Vehicles

• The window is often the most vulnerable part of a vehicle.

• Heavy armour seriously restricts visibility in tanks and helicopters.

• Virtual reality technology can be used to enable the pilot to “see through” the vehicle using cameras.


Tourism & Education


Archaeology

• Can overlay present and past with augmented reality to help tourists and students learn more about a certain historical event.

• Imagine walking onto a Civil War battlefield and seeing a re-creation of historical events on a head-mounted, augmented-reality display.

• It would immerse you in the event, and the view would be panoramic.

19


Augmented Reality & Archaeology


Online Communities


Online Virtual Worlds

• Number of online multiplayer games use virtual reality.

• Geographically distributed people interact within the same virtual world.


Second Life

• Internet-based virtual world launched in 2003, developed by Linden Research, Inc.

• A downloadable client program called the Second Life Viewer enables its users, called "Residents", to interact with each other through motional avatars.

• Residents can explore, meet other Residents, socialize, participate in individual and group activities, create and trade items (virtual property) and services from one another.


Second Life


Second Life

• Second Life's virtual currency is the Linden Dollar (Linden, or L$) and is exchangeable for real world currencies in a marketplace consisting of residents, Linden Lab and real life companies.

• 11.5 million registered accounts.

• Similar to MySpace etc. in that the users are the main generators of content.

• See: http://secondlife.com/

20


Commercial Aspects of Second Life

• Linden Labs makes money by selling land.

• Second Life enables companies to:– Receive product feedback from customers around the

world.

– Meet with global partners at virtual headquarters.

– Discover a new world of online fundraising capabilities.

– Build an island where customers and clients can test out new designs and concepts.

– Build community around branding.

– Real-time 3-D collaboration instead of conference calls.


Commercial Aspects of Second Life

• Companies have also been set up to promote real world businesses in Second Life.

• For example, to organise the creation of a virtual island.

• Promote events in Second Life.

• Train companies to use Second Life


Business in Second Life

• Linden dollars can be exchanged for real dollars.

• So many companies have been set up to make money in Second Life.

• For example, companies create houses, buildings, terrains, avatars, clothes, or an entire island.

• Organise auctions of land, property etc.

• Casinos, bars, etc.

• Any business in real life can be carried out in Second Life.


Challenges and Future Directions


Challenges and Future Directions

• Currently many hardware problems with virtual reality.

• Virtual reality technology will become more popular as these are overcome.


Virtual Reality and Robotics

• Close link between virtual reality and robotics.

• Convincing VR depends on actuators etc, which need much more development.

• Telepresence also depends on sophisticated robots.

21


Internet2

• Tele-immersion needs faster Internet speeds.

• Internet2 is a consortium trying to create new Internet technologies that may eventually replace the current Internet infrastructure.

• This new network will have a higher bandwidth and speeds 1,000 times faster than today's Internet.

• This high-bandwidth, high-speed network would be better able to transfer the large amounts of data that tele-immersion will produce.


Time to Create Virtual Worlds

• Another major limitation of virtual reality is the time that it takes to create a virtual world.

• The more realistic the environment, the longer it takes to make it.

• User generated content, such as Second Life, may be a partial solution to this problem.


Ergonomics

• Many systems rely on hardware that encumbers a user or limits his options through physical tethers.

• Without well-designed hardware, a user could have trouble with his sense of balance or inertia with a decrease in the sense of telepresence

• Users can also experience cybersickness, with symptoms that can include disorientation and nausea.


Other Future Developments

• Better tracking systems

• More natural ways to allow users to interact with a virtual environment.

• Faster computers - for a wearable augmented reality system, there is still not enough computing power to create stereo 3-D graphics.


Conclusions

• Virtual reality technology can be used to:

– Create immersive virtual environments.

– Give the illusion of presence at a remote location.

– Allow people to collaborate within a combination of real and virtual space.

– Overlay information over the real world.


Conclusions

• Many existing applications of virtual reality.

• As the technology improves, it will be used more and more.

• Exciting developments and business opportunities with online virtual worlds.

22


Resources


Resources

• Overview of virtual reality: http://www.howstuffworks.com/virtual-reality.htm

• Site dedicated to virtual reality: http://vroot.org/

• Tele-immersion overview: http://electronics.howstuffworks.com/holographic-environment.htm

• Retinal scanning display: http://seattlepi.nwsource.com/business/27731_retina18.shtml


Virtual Reality and Medicine

• Virtual reality and medicine: http://www.ynl.t.u-tokyo.ac.jp/~nakamura/Economist_com.htm

• Robotic surgery: http://electronics.howstuffworks.com/robotic-surgery.htm

• Electronic books for surgical training: http://graphics.cs.brown.edu/research/telei/teleImmersionApplications.html


Virtual Reality and Medicine

• Da Vinci surgical system: http://www.intuitivesurgical.com/products/davinci_surgicalsystem/index.aspx

• Augmented reality corrects tunnel vision: http://technology.newscientist.com/article/dn9886-augmented-reality-glasses-tackle-tunnel-vision-.html

• Augmented vision for visually impaired: http://www.eri.harvard.edu/faculty/peli/projects/head-mounted.html

• Virtual reality therapy company: www.virtuallybetter.com


Resources

• Augmented reality version of Pacman: http://www.newscientist.com/article/dn6689-human-pacman-hits-real-city-streets.html

• America’s Army: http://www.americasarmy.com/• Article on America’s Army:

http://www.washingtonpost.com/wp-dyn/content/article/2005/05/26/AR2005052601505.html

• Second Life: http://secondlife.com/• List of businesses and organizations in Second

Life: http://en.wikipedia.org/wiki/Businesses_and_Organizations_in_Second_Life

iua awk11 virtualreality - david gameziua.davidgamez.eu/documents/iua_awk11_virtualreality.pdf ·...

Documents