A Pressure Sensing Floor for Interactive Media Applications

Prashant Srinivasan David Birchfield Gang Qian Assegid Kidané Arts, Media & Engineering and Department of Electrical Engineering

Arizona State University, 699 S. Mill Ave. Suite 395 Tempe, AZ 85281, USA

{p.srinivasan, dbirchfield, gang.qian, assegid.kidane}@ asu.edu

ABSTRACT This paper explores the design of a reconfigurable large-area high-resolution pressure sensing floor to help study human dance movement. By measuring the pressure of a user interacting with the system, our device is able to provide real-time knowledge about both the location of the performer on the floor as well as the amount and distribution of force being exerted on the floor. This system has been designed to closely integrate and synchronize with external systems including marker-based motion capture systems, audio-sensing equipment and video-sensing technology, thus allowing for robust multimodal sensing of a subject in the integrated environment. Furthermore, the mats comprising the floor can be readily re-arranged in order to allow for a large number of configurations. Some other possible applications of the pressure sensing floor include virtual reality based entertainment systems or video game control interfaces as well as rehabilitation projects for disabled people with foot or motor-control disorders.

Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation]: User Interfaces

General Terms Design, Experimentation, Human Factors

Keywords Tangible interfaces, pressure sensing, foot sensing, multimodal sensing, networked floorspace, interactive technology.

1. INTRODUCTION Pressure sensing offers a major enhancement for applications

that track and make use of human interaction with computer systems. Recent focus has been on interaction with systems based on observation of gestures or movement by augmenting the user�s environment with novel sensors. For instance, Sony Computer Entertainment introduced the �EyeToy� camera device [13] for enhancing gaming based on gesture recognition. Along these

lines, we have developed a portable pressure sensing floor prototype that is capable of detecting and displaying pressure information about subjects present in the sensing environment. It is constructed of modular light-weight high resolution pressure sensing mats that are readily reconfigurable, allowing creation of floorspaces of varying sizes and arrangements. It transmits detailed pressure information to a host computer over high-speed Ethernet connections, and this information is collated and collectively displayed. A key aspect of our design is the capability of this system to be integrated with existing marker and vision based sensing systems to provide robust multimodal sensing capabilities. Major benefits include high density of sensors allowing accurate representation of interacting objects as well as ease of reconfiguration to suit external environments. Domain experts from dance and kinesiology are currently evaluating the prototype in order to assess its sensing capabilities. The system will initially be used to study interactive dance movement. The prototype could also supplement a videogame gamepad by interfacing to devices like a PlayStation console. This would add an extra dimension to how one interacts with videogames. Foot pressure activity detected during jumping, walking or running could be combined with foot orientation and location information to implement modifications to regular games, allowing intuitive game play such as kicking or blocking in a fighting game.

In related prior work, various systems have been designed to capture and view pressure information associated with people moving across a floor. The surveyed systems only have sensor densities capable of providing pressure distribution information for a small area and cannot fit in the tight space available under a typical dance floor. Also, these lack the capability to form a major part of an integrated environment involving other sensing technologies based on vision, sound and motion. Some approaches [1, 9] make use of load cells set underneath the tiles to measure weight and pressure distributions. However they cannot identify and produce detailed pressure data for an object, losing the ability to report on the location, orientation or resolution of the feet or other objects interacting with the floor. VTT Information Technology�s floor system [6] has only 196 sensors in all and this sensor resolution would not be suitable for dance or movement disability studies that we are targeting. The Magic Carpet [10] from MIT lacks spatial sensor resolution due to the 100 mm distance between adjacent sensors resulting in large zones that do not detect any pressure. The LiteFoot prototype [5] from the University of Limerick has over 1900 optical proximity sensors on the large 1.76 square meter floor area, but spatial resolution of 40 mm does not meet the demands of our targeted applications such as dance analysis. Also, force measurements

from a single accelerometer instead of individual pressure sensors form the basis for the pressure information in this work. A floorspace using Z-tiles [12] was developed to have the ability to form a pressure sensitive area of varying size and shape. This floor makes use of small interlocking tiles that have the ability to network. However, there are only 20 force sensitive resistors in each tile, which does not allow collection of very detailed pressure data in real-time. In-shoe sensors have been considered for the force and pressure feedback of human movement [11]. But they offer limited feedback as data is only collected from the foot. Also, location and orientation information of the foot with respect to the floor is not captured. In our previous work, we developed the ISAfloor, a precursor project to the work described in this paper, which was a smaller prototype floor with 256 force sensing resistors. This floor was developed and tested at a scan rate of 10 Hz. During tests, we noticed that the ergonomics of the foot with cavities between the toes and elsewhere created various �dead zones� with no sensor information available during. This formed the motivation to design an easily reconfigurable floor with a denser array of sensors and higher scan rate for applications requiring more detailed pressure information in real time, such as walking pattern analysis for rehabilitation.

2. THE PRESSURE SENSING FLOOR The prototype consists of several sensor mats capable of gathering pressure information, microcontrollers, Rabbit Ethernet-enabled controllers and a Windows based application program for data visualization, data recording and playback. The mats are grouped together to form a cluster that sends real-time data to the host computer, and can be synched to an external signal propagated through the Ethernet connection of the mats to the host computer. Figure 1 shows the overall system diagram for a cluster or mats.

Figure 1. System Block Diagram for a cluster of pressure

sensing mats

2.1 Floor Hardware and Software The system design in progress is one that can be configured with as few as two mats having a total of 4,032 sensors covering an active area of about 4,160 square centimeters, or as many as 128 mats with a total of over 258,000 sensors and an area of about 26.7 square meters. The sensor elements of the mat are made using a pressure sensitive polymer between conductive traces on sheets of Mylar. Calibration of the system involves application of known and controlled pressure for certain durations of time that can then be used as references. Each sensor has an approximate active area of 6 mm x 6 mm, and is sufficient for contemporary pressure measurement systems for human movement as suggested

by Urry [14] and Davis, et al. [2]. The sensor mat consists of a 42 x 48 grid of pressure sensors on a single model-5315 mat from Tekscan [7]. The dimension of each mat is approximately 62 cm x 53 cm, taking into account the connection tab. The active area is 48.8 cm x 42.7 cm. The resistance of the pressure sensitive paint is of the order of Mega ohms in the absence of pressure and drops to a few kilo ohms when pressure is applied. The sensor mat is rated at 30 pounds per square inch (PSI) or about 207 kilopascals, and best accuracy is achieved for a range of about 14 kilopascals to 207 kilopascals. The following figure (Figure 2) shows the system overview for a single mat.

Figure 2. System Block Diagram for a single mat

In order to create a seamless sensing surface from numerous individual mats, the pads are built in four slightly varying ways, providing overlaps for the non-active border areas. This is accomplished by designing the wooden frame of the mat to be one of four different specifications. The mat protrudes over the frame in certain frame types and vice versa in order to exactly fit over the non-sensing border area of the adjoining mat and thus allowing for the pressure floor to capture and measure motion of pressure across the individual mats of the floor. The following figure shows a sensor mat and associated hardware circuitry.

Figure 3. Pressure Sensing Mat and Controller Circuitry

Each mat is connected to a processing and control board that schedules data scanning and facilitates collection of the scanned data to be transmitted. Timing and control signals are generated by a PIC microcontroller and this regulates the scanning of the mats at 30 Hz. A high-speed analog to digital converter translates the sensed pressure value to an 8-bit digital data that is transferred to a single-board Rabbit computer with Ethernet through a data

port after interrupt-enabled handshaking. A complete frame of data consisting of information from all sensors is collected by the Rabbit board and transmitted to the host computer via an Ethernet link and a high-speed switch. This way, many individual mats can be easily connected to the host computer. Software on the host collects data from the Rabbit board and displays it, indicating the current pressure levels at various points on the selected mat.

2.2 Integration for Multimodal Sensing Multimodal systems have been demonstrated to be more effective and robust than independent systems because humans assimilate and absorb information better when communication occurs through multimodal information channels. Currently, the Interdisciplinary Research Environment for Motion Analysis (IREMA) at Arizona State University aims to create a multimodal sensing and feedback environment for human motion analysis. It houses a marker-based 16-camera system from Motion Analysis Corporation and a 16-channel wireless TeleMyo� 2400 EMG system from Noraxon. Data captured from multiple sensing devices will be synchronized to enhance data fusion. An overview of the multimodal sensing environment is shown in Figure 4.

Figure 4. Overview of multimodal sensing environment

By putting the marker-based motion capture system, video cameras, the EMG system, an audio-based sensing system, and the pressure sensitive floor within a single motion capture space, we will be able to perform a holistic capture and analysis of human movement. We would like to establish statistical models such as Bayesian beliefs networks [3] to model the underlying relationships among human movement, muscle tension and floor pressure. We hope that these models can be used to train video cameras to produce motion capture that is cheaper and yet have performance comparable to that of the marker-based system. Previously, dance scientists have utilized pressure and floor plates to measure a variety of dance-related issues such as landing parameters of various jumps, weight distribution and changes in these aforementioned measurements with alterations in shoe design [4, 8]. Many studies use pressure-based methods to investigate causes of dance related injuries such as shin splints, stress fractures and knee pain. A detailed pressure-based high-resolution image of foot or body interaction with the floor prototype coupled with video or marker-based information about the dancer�s movement will allow for further research into dance movement along these lines. Also, such multimodal information could be used to design virtual space walkthroughs, where

activity like walking or running could be monitored by the floor, and video information could be used to project relevant images using computer displays. Initial work on integration of the floor with other sensing technologies has involved the successful implementation of a signal mechanism to synchronize the floor with an external clock signal propagated to all the pads via the Ethernet connection to the host computer. This allows us to send a synchronizing clock signal to all pressure pads simultaneously, and this is used to synchronize the scanning of each mat.

2.3 Preliminary Results Evaluations on the prototype system consisting of two pressure sensing mats connected to the computer have been completed. Following tests with human subjects as well as objects, and informed by feedback from domain experts in dance and kinesiology, we note that the designed floor provides reliable pressure data. It is robust and easily reconfigurable. The pressure mats can be calibrated to display pressure readings in PSI, but as we are interested currently in measuring relative pressure we only collect the raw 8-bit digital data value proportional to the local pressure being applied on the mat. We have succeeded in dynamically reassigning network identification keys, thus making the system �plug-and-play� as well as capable of broadcasting the pressure information over our network to remote workstations. Consistent with our objectives, a scan rate higher than 30 Hz for the sensors has been achieved without using any hardware or software compression techniques and we will seek to boost this performance through hardware and software revisions allowing dynamic compression. Importantly, our marker-based motion capture systems run at 60 Hz, and real time data from the floor at 30 Hz can be easily synchronized to that of the motion capture system. Also, this allows for collection of data at a rate sufficient for expressing human foot movement that could be used to control algorithms or other hardware devices. We were able to interact with the floor and use it to detect in real time the pressure points on the foot during the walk cycle. We observe that the current scan rate allows us to clearly discern the shift in foot pressure during various stages of the human gait cycle from the heel to the ball of the foot and then to the toes. When compared to previous work, the existing rate of this system compares favorably and the large number of sensor elements scanned gives the advantage of detailed data capture. Below, shifting of pressure from the heel towards the ball of the foot, and then the toes is clearly visible.

Figure 5. 2-D representation of the detected pressure pattern

for a foot during a walk cycle

The pressure patterns from a foot are well detected by the system, and clearly reconstructed at the host computer. The high density of sensors allows for this clear pattern detection. The system successfully demonstrated the ability to synchronize scanning and transfer of data to an external clock, received through the Ethernet connection. The pressure map below illustrates the distribution of pressure captured with a subject standing on the prototype.

Figure 6. 3-D view of a pressure map for a human standing on

the pressure sensing floor

3. CONCLUSIONS We have successfully designed and implemented the

software and hardware architecture of a high-resolution pressure sensing floor. Currently, we have implemented a full cluster consisting of two mats and we have been able to accurately detect pressure data from the pads simultaneously connected to the host computer via Ethernet. This system architecture is robust and provides complete real-time data about pressure and relative location as well as orientation and movement on any activity that takes place on the prototype. This floor system can be easily extended by interfacing several more mats identical to the ones currently being used by the system, and network all of them to the host computer. We are evaluating a number of hardware and software-based algorithms for data compression to reduce the amount of data being sent over the network in order to dynamically increase or decrease the scan rates for individual pressure sensor pads as per the sensitivity needs of the applications that will use the system. Performing artists such as a musicians and dancers will provide an ideal test bed for further development and evaluation of body sensing technologies because of their highly stylized movements and expertly trained techniques. Also, the pressure data collected along with motion-capture information from the camera based system will provide a means for performing a unique and novel analysis of human foot movement, and this information can be used to drive various interfaces, devices or displays, such as video games, virtual walkway displays and interactive dance floors.

4. ACKNOWLEDGMENTS The work described in this paper is supported under NSF Grant #0403428. We also thank Tekscan and Microchip for their continuing support.

5. REFERENCES [1] M. Addlesee, A. Jones, F. Livesey, and F. Samaria, �The

ORL Active Floor,� IEEE Personal Communications, 4(5), pp. 35-41, 1997.

[2] B. Davis, R. Cothren, P. Quesada, S. Hanson, and J. Perry, �Frequency content of normal and diabetic plantar pressure profiles: implications for the selection of transducer sizes,� Journal of Biomechanics, Vol. 29, No. 7, pp. 979-983, 1996.

[3] R. O. Duda,. P. E. Hart, and D. G. Stork, �Pattern Classification�, Wiley Interscience, New York, Second edition, 2001.

[4] A. Gans, �The relationship of heel contact in ascent and descent from jumps to the incidence of shin splints in ballet dancers,� Physical Therapy, Vol. 65, No. 8, pp. 1192-1196. August 1985

[5] N. Griffith and M. Fernström, �LiteFoot: A floor space for recording dance and controlling media,� In Proceedings of the 1998 International Computer Music Conference, International Computer Music Association, San Francisco, U.S.A., pp. 475-481, 1998.

[6] J. Leikas, J. Mattila, L. Cluitmans, and T. Urhemaa, �IMS Intuitive Movement Sensing Method,� In Proceedings of the Smart Objects Conference, Grenoble, France, pp. 200-203, May 2003.

[7] C. F. Malacaria, �A Thin, Flexible, Matrix-Based Pressure Sensor,� Sensors Magazine, Vol. 15, No. 9, pp. 102-104, September 1998.

[8] C.D. Miller, L.E. Paulos, R.D. Parker, and M. Fishell, �The ballet technique shoe: a preliminary study of eleven differently modified ballet technique shoes using force and pressure plates�, Foot Ankle, Vol. 11, No. 2, pp. 97- 100. October 1990.

[9] R. J. Orr, and G. D. Abowd, �The Smart Floor: A Mechanism for Natural User Identification and Tracking,� Extended Abstracts of the Conference on Human Factors in Computing Systems (CHI 2000), The Hague, Netherlands, pp. 275-276, 2000.

[10] J. Paradiso, C. Abler, K. Hsiao, and M. Reynolds, �The Magic Carpet: Physical Sensing for Immersive Environments,� Ext. Abstracts CHI 1997, ACM Press, pp. 277-278, 1997.

[11] J. Paradiso, K. Hsiao, A. Benbasat, and Z. Teegarden, �Design and Implementation of Expressive Footwear,� IBM Systems Journal, Volume 39, Nos. 3 & 4, pp. 511-52, October 2000.

[12] B. Richardson, K. Leydon, M. Fernström, and J. Paradiso, �Z-Tiles: building blocks for modular, pressure-sensing floorspaces,� Extended Abstracts of the 2004 conference on Human factors and computing systems, Vienna, Austria, pp. 1529-1532, 2004.

[13] Sony Computer Entertainment, Eyetoy� website, Accessed 28 April 2005, url: http://www.eyetoy.com/

[14] S. Urry, �Plantar pressure-measurement sensors,� Measurement Science and Technology, IOP Publishing Ltd, Bristol, UK, Vol. 10, No. 1, pp. 16-32, 1999.