X R D S • F A L L 2 0 1 5 • V O L . 2 2 • N O . 130

feature

Imag

e by

Alic

ia K

ubis

ta /

And

rij B

orys

Ass

ocia

tes

/ S

hutt

erst

ock.

com

Using Your Own Muscles: Realistic physical experiences in VR Leveraging the user’s own muscles to simulate impact and forces from a virtual reality world allows us to create more immersive experiences without bulky equipment.

By Pedro Lopes, Alexandra Ion, and Robert KovacsDOI: 10.1145/2810243

C onsumer virtual reality headsets (such as the Oculus Rift, Samsung Gear VR, or Google’s Cardboard) provide an exciting peek through the window of virtual reality (VR), but that is where our experience ends. As we reach out to touch these novel realities we are left with a sense of disillusion, because all we see are 3-D ghosts

without any mass. To give VR users a sense of an object with mass, traditionally large robots are used that can, for instance, move the user or simulate virtual terrain. However, these large and stationary robots are out of place in the mobile age: We are used to small and light, yet powerful, devices in our pockets. After all, consumer-ready VR became a reality when the headsets finally met the size and portability requirements of our mobile times.

At the Hasso Plattner Institute’s HCI Lab, our research has focused primarily on bringing a physical component to mobile VR experi-ences. Users can reach out and feel what is, virtually, there. Because we have engineered these devices to be wearable, they leave the user uncon-strained to freely move in space and enjoy the virtual experience to its fullest. In this article we describe a new world of VR, in which users feel forces from the virtual world through

their fingers, arms, and even the ter-rain under their feet. This is our con-tribution to making virtual reality physical.

MOBILE TERRAIN SIMULATIONFrom the inception of VR in the 1960s [1], through the industry hype of the 1980s, up until today, many efforts have been geared toward helping users to “feel” virtual worlds. Industry and research labs responded to this chal-lenge by using motion platforms, such

as Lufthansa’s flight simulator.1 These motion platforms are heavy station-ary machinery that are too impractical to ever reach our homes—even if you could afford a large motion platform, space is sacred these days. This tech-nology was acceptable when VR was a specialist product, and these ma-chines shone through their precision rather than their form factor. In fact, the biggest players in VR were not gam-

1 http://www.virtual-fly.com

X R D S • F A L L 2 0 1 5 • V O L . 2 2 • N O . 1 31

X R D S • F A L L 2 0 1 5 • V O L . 2 2 • N O . 132

feature

with just a headset, you encounter vari-ous obstacles that you cannot interact with using your physical senses. Howev-er, with Level-Ups you can actually step on these objects as depicted in Figure 1. To get across obstacles, you physically lift your foot and climb across. In the physical reality, we see physical eleva-tion is rendered using our Level-Up mo-torized stilts. Depicted here, Level-Ups provide the physical sensation of an el-evated ground to a user wearing a head-mounted display tracked by an optical motion capture system. Every Level-Up device consists of a boot, mounted onto a lift table, which in turn is mounted onto a simple artificial foot. The lift ta-ble is actuated by a motor, which is ob-served by an encoder and controlled by a microcontroller that communicates via bluetooth with VR applications. Each boot can render an elevation of up to 12.5 centimeters. Because Level-Ups only extend while the user’s foot is in mid-air, they can be fully extended or contracted with a compact 40W mo-tor in half a second, yielding a wearable free-walking physical feedback device.

THE CHALLENGE OF MOBILE FORCE FEEDBACKLevel-Ups was, for us, a clear step in the right direction toward mobile physical VR, but is it what sci-fi had us dream-ing about for years? What about feeling forces from the virtual world?

These dreams of a physically be-lievable virtual reality aren’t new, but so far they required heavy gear, such as exoskeletons. These are large me-chanical contraptions mounted onto users; a motor is placed at every joint of their limbs and thus users are able to manipulate their limbs while they experience VR. Think of motion plat-forms that you carry around. The hu-man is rather powerful, and therefore the motors required to move our limbs must be powerful, too. And powerful motors are large. Now if you look at Google Cardboard, Oculus Rift, or any other headset on the market, you real-ize their form factor is orders of mag-nitude smaller than the actuator sys-tems that would allow us to bring some physicality to virtual experiences. So how can we produce enough power to actually move a human limb, but in a very concise form factor?

ers, but training centers (e.g., airplane pilots) and amusement parks (e.g., VR rides).

These massive motion platforms were able to provide the sense of “be-ing moved” or the sensation of accel-eration, but weren’t really suited for simulating a “walk in the park.” Thus, many researchers designed mechani-cal devices targeted at rendering the sensation of stepping onto uneven ground. Canonical examples of these were treadmills equipped with indi-vidually height-adjustable elements that simulated bumpy terrain and virtual slopes [2], or the CirculaFloor, which uses four robot units that place themselves under the user’s steps [3]. However, in the era of consumer-ready mobile VR we cannot expect to own a

piece of infrastructure such as Circu-laFloor. It is enough having a Roomba roam around a house for cleaning pur-poses, but four jumbo-sized Roombas moving around the living room for the sole purpose of emulating VR terrain?

In one of our projects, we took a dif-ferent approach to ground simulation. Instead of placing the human into a robotic device, we mounted motors under the human. With this approach, the physical feedback device goes wherever the human goes—it is mo-bile. Our computer-controlled stilts, or “Level-Ups,” allow users to physically experience elevation. Level-Ups are designed to allow users to freely walk in the virtual and physical world, and produce a higher sense of presence.

While you walk in a virtual world

Figure 1. The Level-Up motorized stilts allow users walking in a spatial virtual reality environment to experience physical elevation.

X R D S • F A L L 2 0 1 5 • V O L . 2 2 • N O . 1 33

an Arduino microcontroller, a Blue-tooth modem, and an EMS unit (similar to muscle rehabilitation devices found in a doctor’s clinic), which outputs a maximum 50V/100mA. If you stop and think for a second about these num-bers, you will realize the 100mA needed to actuate muscles is two orders of mag-nitude more efficient than actuating a conventional motor. This happens be-cause, in reality, the electrical muscle stimulation is just a “control signal” to

At first this sounded as contradic-tory as Newton’s Third Law, which de-mands a counter force to every force you want to apply; this is why an exo-skeleton has to be a certain size—it must attach to your arm and apply a counter force (called force feedback) to your muscle motion. The same applies if you want to play virtual soccer: A ro-botic contraption would need to apply a force to your foot every time you hit the massless virtual ball you see on the headset. So how do you keep the required amount of force feedback, but scale down the hardware? Our ap-proach has been to simply take all the mechanics out of the equation and replace them with our own muscles. Because our technique uses human muscles to create the force sensation, we called it “muscle-propelled force feedback.”

LEVERAGING THE USER’S MUSCLESMuscle-propelled force feedback brings force feedback to mobile de-vices. It explores the user’s own muscle power as a replacement for motors. To achieve this, we actuate the user’s muscles using electrical muscle stim-ulation (EMS), a technique first ex-plored in physical rehabilitation in the 1960s and 1970s [4], and more recently in the research of Tamaki, Miyaki, and Rekimoto [5].

Figure 2 illustrates the use of our mobile force feedback prototype in a mobile gaming scenario. The mus-cle-propelled prototype is mounted on the back of a mobile phone. The player connects to it by attaching two electrodes to the flexor muscles on the forearm as depicted in Figure 3. The game requires the user to steer an airplane through strong side winds by tilting the device. The game ren-ders the wind force by tilting the de-vice against the user’s will. It achieves this by stimulating muscle tissue in the user’s arm though the electrodes, triggering an involuntary contrac-tion. This causes the user’s arms to tilt sideways and thus the device to tilt. Since the airplane is controlled by tilt, the involuntary tilting threat-ens to derail the airplane. To stay on course, players counter the actuation using the force of their other arm. As we showed in a study, participants

perceived the interaction between the involuntary and voluntary tilt as force feedback. In fact, these participants rated muscle-propelled force feed-back as more exciting than vibration feedback, which is the standard feed-back modality for mobile devices.

The great thing about not using mo-tors is we were able to miniaturize the device to fit in a 133 mm × 70 mm × 20 mm casing, weighing only 163 grams. This simple prototype is comprised of

Figure 2. Our prototype electrically stimulates the user’s arm muscles via the electrodes shown, causing the user to involuntarily tilt the device. As this user is countering this force, he perceives force feedback.

Figure 3. A close-up of our prototype and the electrodes, which carry the medically compliant electrical signals that actuate the user’s muscles.

X R D S • F A L L 2 0 1 5 • V O L . 2 2 • N O . 134

feature

We describe a new world of VR, in which users feel forces from the virtual world through their fingers, arms, and even the terrain under their feet.

store in your basement.This is why the most conventional

way to emulate the sensation of impact is to use vibration. There are haptic suits out there, such as the Tesla Suit,2 which has embedded vibrotactile and electro-tactile stimulators touching your skin. This is a great step forward, however Newton would certainly not be thrilled about it since vibration motors only stimulate the skin’s surface and will not make your arm move. An impact sensa-tion is more complex than you might think. For instance, if someone pushes you, your skin detects pressure through its mechanoreceptors at the location you were touched, your muscles move and your proprioceptive sense will tell your brain your muscles are being moved. So the requirements for impact emulation pile up: We need to stimulate the skin, we need to apply pressure, and we need to inform the proprioceptive sense by moving your muscles. We need to do all of this, and keep the hardware smaller and less dangerous than a mas-sive robotic arm. The answer is, again, your own muscles.

To tackle this challenge we built impacto, a device designed to render the haptic sensation of hitting or be-ing hit in virtual reality as shown (see Figure 4). The key idea that allows our rather small device to simulate a mas-sive hit is it decomposes the stimulus into two distinct components: tactile sensation and force feedback sensa-tion. Tapping the skin with a solenoid renders the tactile sensation; the force of the impact is added by thrusting the user’s arm backwards using elec-trical muscle stimulation. In fact, because the EMS is able to move the arm at such a compact form factor, we can keep the solenoid component to a minimum size, because all it needs to do is tap the skin. As a matter of fact, participants of our study rated impact simulated using impacto’s combina-tion of solenoid hits and electrical muscle stimulation as more realistic than either technique in isolation.

The sensation of impact plays a key role in many sports simulations such as boxing, fencing, football, etc. Imag-ine all the sport simulators, games, and digital experiences you had so far,

2 http://www.teslasuit.com/

the muscle, and the body itself has a large supply of energy that is used to ac-tually perform the contraction.

After we demonstrated muscle-pro-pelled force feedback at a few confer-ences (ACM’s Augmented Human’13, ACM CHI’13, and IEEE World Hap-tics’13), some of our peers were really excited about the idea of having mo-bile force feedback. While this type of force feedback works well for “invisi-ble” forces like wind or magnetism, we started asking the next question: How would this feel in virtual reality when a visible object hits you? Can we create a believable physical sensation? Can we let the user feel the virtual world?

REALISTIC PHYSICAL EXPERIENCES IN VRIt turns out this is harder than it sounds. When somebody hits you in “real” physics-based reality, you feel the point of contact and you feel the impact

transmitted onto your limbs. So how can we replicate that sensation? One possibility is to add a robotic arm that hits you in the “physical world” exactly when you get hit in the “virtual world.” Unfortunately that is (1) dangerous as it might break our bones into pieces, (2) expensive, and (3) requires large ma-chinery that you likely don’t want to

Figure 4. Impacto is designed to render the haptic sensation of hitting and being hit. The key idea that allows the small and light impacto device to simulate a massive hit is it decomposes the stimulus into tactile feedback (rendered by a solenoid) and force feedback (rendered by electrical muscle stimulation).

X R D S • F A L L 2 0 1 5 • V O L . 2 2 • N O . 1 35

the consumer form factor: mobile and wearable. This is the start of the end of virtual experiences with no physical sense of force, impact, and terrain eleva-tion. Companies have been marketing their VR products as “experiences like you are actually there” (e.g., Google’s 360 camera rig) or “feel the sudden im-pact of a bullet” (e.g., the Tesla Suit), but these are ghost experiences without any strong physical sensations. However, by actuating the users’ own muscles we can provide strong physical sensations and fit all of this in a convenient mobile package that users have come to expect from technology. Our approach prom-ises a future of physical VR without wearing bulky apparatus, such as large motion platforms or exoskeletons, an approach that leaves the user untangled and free to move.

References

[1] Iwata, H., Yano, H., and Fukushima, H. CirculaFloor: A locomotion interface using circulation of movable tiles. Computer Graphics and Applications, IEEE 25, 1 (2005), 223–230.

[2] Noma, H., Sugihara, T., and Miyasato, T. Development of ground surface simulator for Tel-E-Merge system. In Proc. VR ‘00. IEEE, 2000, 217–224.

[3] Strojnik, P., Kralj, A., and Ursic, I. Programmed six- channel electrical stimulator for complex stimulation of leg muscles during walking. IEEE Trans. Biomedical Engineering BME-26, 2 (1979), 112–116.

[4] Sutherland, I. E. A head-mounted three dimensional display. In Proc. AFIPS ‘68. ACM, New York, 1968, 757–764.

[5] Tamaki, E., Miyaki, T., and Rekimoto, J. Possessed hand: Techniques for controlling human hands using electrical muscles stimuli. In Proc. CHI ‘11. ACM, New York, 2011, 543–552.

Biographies

Pedro Lopes is a Ph.D. student of Prof. Patrick Baudisch’s Human Computer Interaction lab in Hasso Plattner Institut. Lopes creates wearable interfaces that read and write directly to the user’s body through our muscles [aka proprioceptive interaction]. He augments humans and their realities by using electrical muscle stimulation to actuate human muscles as interfaces to new virtual worlds. He also enjoys playing improvised music and is the digital content editor of XRDS.

Alexandra Ion is a Ph.D. student of Patrick Baudisch at the Human-Computer Interaction lab at the Hasso Plattner Institute. She is interested in haptics and mechanical properties in fabrication. Ion has previously worked with Dr. Michael Haller at the University of Applied Sciences Upper Austria, Campus Hagenberg on large interactive whiteboards with pen input.

Robert Kovacs is a mechatronics engineer and artist. Currently, he is a Ph.D. student at the Human-Computer Interaction lab at Hasso Plattner Institut and works on large-scale fabrication. He previously studied at Free University in Berlin and the University of Novi Sad, Serbia. Kovacs is the co-founder of Oktopod Studio, an educational tool for mechatronics, robotics, and automation based on open-source hardware.

© 2015 Copyright held by Owner(s)/Author(s). Publication rights licensed to ACM.

1528-4972/15/09 $15.00

now image that all those impacts be-come real, and you feel them in your skin and muscles.

To demonstrate how one could feel impact in VR experiences, we imple-mented three VR sport simulators: boxing, soccer juggling, and baseball.

Boxing is a sport in which impact sensation is crucial. The opponent’s av-atar keeps its guard up and attacks pe-riodically. Users must choose the right moment to start a successful attack. When the virtual character attacks, the impact of the virtual punch is rendered with impacto—users feel the tactile sensation on the location where the avatar hits them and they feel their arm being thrust backwards due to the mass of the avatar’s arm colliding with their own arm. When users attack, again we render the impact by tapping users at the location they encountered the oppo-nent, e.g. their fist or arm, and stimulate their biceps causing their arm to stop upon collision.

Users can use multiple impacto units at the same time, as well as mounting them on other limbs and muscles (such as the triceps, quads, etc.). In Figure 5 we mounted a unit to the user’s calves, which enables a different VR experi-

ence: soccer juggling. This setup points the solenoid component at the user’s in-step (top of the foot) and the EMS unit to the calf muscles. The electrical muscle stimulation causes the foot to slightly bend backward at the moment the ball hits the foot. Also, by placing impacto units on arms and legs, users experience a Thai boxing simulator where they feel virtual punches and kicks. As in all our sport simulators, we use a Kinect cam-era to track the user’s body posture and an additional wireless accelerometer on the solenoid to determine the foot’s tilt.

Lastly, we can actuate users, even while they operate objects. This allows impact sensation to happen in hand-held props that the user manipulates in a virtual world: Imagine hitting a baseball with your VR bat. Figure 5 shows a prop becoming animated; here a stick is a stand-in for a baseball bat. We mounted the solenoid onto the prop but the EMS unit, in contrast, stayed with the user; here it stimulated the wrist muscles exactly when the ball hits the virtual bat.

CONCLUSIONSWe are finally getting rid of the mass-less ghosts in VR and are sticking to

Figure 5: By wearing impacto on the feet, the user experiences the impact of kicking a virtual soccer ball (left). He hits a virtual baseball with impacto’s solenoid on the prop and electrodes on the wrist extensor muscle (right).