ieee transactions on visualization and computer … · specifically, we used passive haptics to...

1077-2626 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

Manuscript received 10 Sept. 2019; accepted 5 Feb. 2020.Date of publication 18 Feb. 2020; date of current version 27 Mar. 2020.Digital Object Identifier no. 10.1109/TVCG.2020.2973055

Presence, Mixed Reality, and Risk-Taking Behavior:

A Study in Safety Interventions

Sogand Hasanzadeh, Nicholas F. Polys, Member, IEEE, and Jesus M. de la Garza

Fig. 1. Experimental Design

Abstract—Immersive environments have been successfully applied to a broad range of safety training in high-risk domains. However, very little research has used these systems to evaluate the risk-taking behavior of construction workers. In this study, we investigated the feasibility and usefulness of providing passive haptics in a mixed-reality environment to capture the risk-taking behavior of workers, identify at-risk workers, and propose injury-prevention interventions to counteract excessive risk-taking and risk-compensatory behavior. Within a mixed-reality environment in a CAVE-like display system, our subjects installed shingles on a (physical) sloped roof of a (virtual) two-story residential building on a morning in a suburban area. Through this controlled, within-subject experimental design, we exposed each subject to three experimental conditions by manipulating the level of safety intervention. Workers’ subjective reports, physiological signals, psychophysical responses, and reactionary behaviors were then considered as promising measures of Presence. The results showed that our mixed-reality environment was a suitable platform for triggering behavioral changes under different experimental conditions and for evaluating the risk perception and risk-taking behavior of workers in a risk-free setting. These results demonstrated the value of immersive technology to investigate natural human factors.

Index Terms— Mixed-reality, passive haptics, presence, human factors, risk-taking behavior, X3D, construction safety

1 INTRODUCTION

The construction industry is one of the most hazardous industries worldwide, and the hazardous nature of the construction site makes real-world safety studies difficult [1-3]. Previous studies showed that Immersive Environment (IE) safety-enhancement systems have successfully simulated the hazardous conditions of a high-risk environment, such as military and aviation [4-6]. Consequently, the construction industry is increasingly using IEs as an approach for safety learning [2, 7]. Although immersive environments have been

used broadly for safety training [2, 8], few studies used these systems to examine the risk-taking behavior of individuals. However, such systems are ripe for explorations of the Risk Compensation Theory [9], wherein researchers evaluate how implemented safety interventions decrease individuals’ perception of risk, which in turn, prompts more risky behaviors. In the construction industry, we assume that as the level of safety interventions changes, workers will adjust their adaptive behavior. Such risk-compensatory behavior will therefore, highly depend on each individual’s target level of risk and perceived risk [10, 11]—which will both be impacted by the realness of the designed experiment. Consequently, harnessing immersive environments within construction research will yield novel insights into the risk-taking behaviors of construction workers without exposing them to physical risks.

This study focused on falls-to-lower-level hazards since statistics show that fall safety interventions (e.g., providing safety training and compelling the use of personal protective equipment) have failed to fully achieve their safety objectives in preventing this leading cause

• ��

• ��

• ��

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020 2115

2116 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020

of death among roofers in the United States [12, 13]. To examine whether risk compensation plays a role in such fatality and injury rates, this study created an immersive, mixed-reality (MR) environment to provide subjects with risk-free interactions while assessing their risky behaviors under different levels of protection. Specifically, we used passive haptics to examine a worker’s spatial perception of completing a roofing task on a sloped roof, and we varied the provided safety interventions to change risk perception. In parallel, we investigated the role of such human factors as personality traits in determining subjects’ sense of Presence and engagement with the MR environment to confirm whether the environment effectively permits studying worker’s risk-taking behavior. Our methodology and validating results will foreseeably empower project managers to more actively mitigate incidents by providing a platform for identifying at-risk workers and generating personalized training.

2 BACKGROUND

2.1 Immersive Environment and Safety Training Using immersive environments to improve safety has gained popularity in many fields in recent years. Immersive environments (virtual reality-VR, augmented reality-AR, and mixed reality- MR) are used to simulate the hazardous situations in a risk-free setting, for example for acrophobia treatment [14] or emergency evacuation [15]. Therefore, the wide IE continuum serves as a new opportunity for improving the safety training process through interactive training methods that bring a real-life aspect into training scenarios.

CAVE virtual displays provide a relatively risk-free opportunity to re-enact on-the-job hazardous situations safely, experience real-time consequences, learn from mistakes, and improve the situational awareness of individuals. However, limited studies have used this technology for safety training as compared to more affordable techniques (e.g., a head-mounted display). In the following, the application of CAVE in military, mining, security, education, and construction domain are discussed.

The U.S. Army Research Laboratory developed a VR system consisting of a CAVE display and an Omni-Directional Treadmill to provide intensive training in hazardous military missions. The locomotion interface facilitates the coordination of the entire system, and as the trainee walks, runs, or jumps, the CAVE display changes appropriately [4]. Although no data were gathered to measure the effectiveness of this training, the authors took a huge step towards making CAVE a useful system for military training purposes. In another study, the five-sided CAVE was used to train detectives in an active shooter response scenario [16]. For security training, depicting a real scenario is very important. The deputy was equipped with the same equipment that they have on patrol, such as real weapons. The floor was designed to project the sound and simulate vibration of a terrifying high-school shooting. During the experiment, live-action 360-degree videos were played to assess the deputies’ responses and reactions. In some studies of CAVE-based VR environments, improving emergency responses have been examined. For example, Ericson (2007) also used a CAVE-based VR training environment to simulate fire situations to increase children’s understanding of fire safety, and to help children learn to escape in this hazardous situation for which practicing in real life is impossible [15]. In another study, a • ve-sided CAVE system was used to study individual’s risk perception and evacuation behavior during a threatening situation such as a tunnel emergency [17].

CAVE has been used as a safety training tool in construction equipment operation. In the study conducted by Yuen and his colleagues in 2010, forklift truck operation was modeled using CAVE-based VR with which drivers can practice forklift truck maneuvers and operations. To make the simulation as real as possible and enhance safety awareness of drivers, “virtual incidents” are created and visualized. The results of the study showed that operators become more vigilant and skillful in handling hazardous

conditions [18]. In another study within construction safety settings, Perlman et al. (2014) used two training techniques (photographs and construction documents as well as a virtual construction site) to investigate the differences in risk perception of superintendents. The results of their study showed that subjects in the virtual environment identified more hazards illustrating that the virtual environment portrayed hazardous situations better [2]. Taking into account the challenges and lessons learned from previous VR safety-related studies, in the present study, we have decided to use CAVE to create mixed-reality to monitor risk-taking behavior of workers.

2.2 Haptic Feedback and Human Spatial Perception Incorporating realistic haptic feedback in human-computer interaction is essential to immersive environments. There are two types of haptic interfaces: active and passive. Active haptics is computer-generated, while passive haptics provides feedback by simulating the shape, weight, or other physical properties of objects [19]. The use of passive haptics in combination with an active device could enrich interactions with virtual environments and significantly improve the user’s spatial perception and performance in a given task [20, 21].

One way to generate a sense of force/tactile feedback for the user in the IEs is to use active feedback hardware devices (such as force feedback gloves) [22]. However, the user needs to be attached to the device even when there is no force feedback, which makes the interaction unusual; and affects human-computer interaction in many ways; for example, the user’s sense of immersion is reduced significantly [21-24].

Contrary to active haptics, which relies on actuators driven by computer systems, passive haptics relies on real, physical objects or props that simulate the touch and force channels and enhance the user’s experience through the existence of a real physical object [21]. A disadvantage is that the passive haptic feedback cannot change during the simulation in real-time [25]. Martin and his colleagues recreated a mixed-reality to achieve realistic movements of a worker during completion of an assembly-like scenario, while movable props representing an object were adjusted to simulate the weight of the actual object [21]. The results of their study showed that simulating the form and weight of actual objects is the key to improve the VR experience of the participants. In one of the recent studies, Nagao and his colleagues instead of simulating a sense of vertical transportation using complicated systems or by using actuators, they have used passive haptic stimuli (i.e., small bumps under the feet of users) to simulate the edges of stairs in a virtual environment [26]. The results of their study showed that the small bumps as passive haptics enhanced the user’s feeling of Presence and the sense of ascending stairs.

Although the virtual-reality has been commonly used in the construction and civil engineering field, there are very limited studies that use immersive graphics and passive haptics to improve the safety learning. For example, for simulating the scaffolding system in a high-rise scenario, Shi and his colleagues used a wood plank in the real-world as a passive haptic combined with a virtual environment to provide workers with a greater tactile realism and sense of height [27]. Therefore, the previous literature made substantial efforts to show that passive haptics can improve the user’s spatial perception and experience in immersive virtual environments.

2.3 Presence in Immersive Environments Although IEs have been used for a wide variety of applications, one of the most important challenges is to determine the effectiveness of IEs, which has driven much research in this area [28-30]. The credibility of an experiment scenario in an immersive environment demands to model true-to-life events and user interactions [31, 32]. One of the common measures of IE effectiveness is the Presence concept, which was first established by Akin and his colleagues in 1983 [33]. Presence is a cognitive parameter and a subjective

phenomenon describable as the sensation of being in an MR environment [34]. They defined Presence as the feeling of “being there” in a virtual place. Witmer and Singer defined Presence as “a subjective experience of being in one place or environment, even when one is physically situated in another” [35].

Presence as a normal awareness phenomenon requires the direct attention of subjects. Involvement is encouraged, and immersion is enabled when there is a close interaction between an environmental factor and sensory stimulation. In fact, people experience various degrees of Presence when they are immersed in an IE, and this determines the extent to which they become involved in the situation portrayed [35]. The two orthogonal components of Presence contribute greatly to the realistic response of participants when they are immersed in a VR system [32]. Place Illusion (PI) means “being there” or “having a sensation of being in a real place” [32], p. 3549; and corresponds to the traditional conception of Presence. Plausibility Illusion (Psi) refers to the illusion that “the scenario being depicted is actually occurring” or believing in what is happening [32], p. 3549. PI is constrained by the virtual reality system properties in terms of its sensorimotor constraints, and Psi is impacted by the credibility of the situation as portrayed compared to expectations. It must be emphasized that, when PI breaks, because it is a perceptual illusion, it can be recovered very quickly, but when Psi breaks, it is unlikely to be recovered.

Previous literature determined that Presence is a multi-component construct affected by both technological factors (i.e., vividness) and human factors (e.g., subject personality, cognitive style, computer experience) [36-38]. In terms of technological factors, in the present study, if subjects experience the sensation of being on the roof of a two-story residential building and participate in the roofing scenario in an authentic way, the risk they perceive will resemble what they might experience when completing the same scenario in the real world. Thus, by combining passive haptics with a projection-based virtual environment, the mixed-reality triggers sensations of height as well as working and walking up and down a roof, despite the fact the subject only walks on a sloped surface built on floor level.

Since the Presence construct is subjective -a qualia (i.e., a subjective and internal feeling elicited by sense perceptions)- with no direct way to measure it, this construct has been measured in many different indirect ways in the literature [32]: self-reporting, physiological, and psychophysical/behavioral measures. All existing measures have advantages and disadvantages. The predominant method for measuring Presence is the use of post-trial self-reporting questionnaires. Skarbez and his colleagues reviewed the use of the most popular self-reporting questionnaires and found that these questionnaires are easy to apply because the questions are general and do not require substantial modification for various types of research [39].

Previous literature also showed that the use of physiological and behavioral responses as surrogates for Presence is methodologically and practically sound [40, 41]. Accordingly, if a person feels that s/he is in the roofing scenario depicted in the immersive mixed-reality environment, then s/he should exhibit changes in behavior and physiological responses concomitant with the fall-risk level. Therefore, this study was designed to elicit measurable physiological (i.e., heart-rate variability) and behavioral (i.e., changes in risk-taking behavior) responses under various experimental conditions.

2.4 Risk Compensation Theory The history of risk compensation (or risk homeostasis) dates back to two pioneering theories by Peltzman (1975) and Wilde (1982) [42], [9]. They argued that the benefits of a decreased risk of an injury associated with providing safety intervention are reduced significantly as a result of increased risk-taking behavior. Both theoretical backgrounds suggested that many safety technological advances and safety interventions may not be optimally effective due to this effect. Although the risk-compensation phenomenon has been studied in various disciplines, the previous studies mostly involved

participants who were never exposed to any life-threating decision-making during the experiment, and their risk-compensatory behaviors were assessed based on self-report ratings. For example, Feng and his colleagues used common self-reported ratings to imagined situations for examining risk-compensatory behavior in the construction industry [43]. Subjective measurement tools are not most appropriate for measuring risk-taking behavior in a high-risk industry such as construction, so, verbal responses to hypothetical scenarios cannot reflect the actual risk-taking behavior. While their study contributes to the body of knowledge, they could not find signs of compensatory behaviors. A proper study of the risk compensation effect requires to more objectively and empirically investigate the effect of safety interventions on individuals’ risk-taking behaviors. The present study suggested using a mixed-reality environment with passive haptics to capture naturalistic risk-taking behavior of workers and to study risk compensation effect in the construction industry. If the developed MR environment provides a higher sense of presence, and in turn, prompts individuals to adjust their risk-taking behavior in response to preventive interventions in accordance with the risk compensation theory; the proposed MR approaches can be considered as one of the promising methods to objectively study the workers’ risk-taking behavior affected by human factors.

3 POINT OF DEPARTURE

This study aimed to examine the undesirable latent side-effects of safety interventions, known as risk compensation. To address this objective, we changed the safety features in the experimental scenario within the mixed-reality environment to assess any associated changes in workers’ risk perceptions, risk-taking behaviors, and successive decision-making. In this paper, we evaluated whether mixed reality is a suitable platform for assessing risk perception and the risk-taking behaviors of construction workers. Specifically, we posited that if simulating height and virtual falls using passive haptics in a mixed-reality environment stimulate sufficient Presence to trigger subjects’ behavioral changes under different experimental conditions, then the MR platform will capture workers’ naturalistic risk compensatory behaviors within a risk-free setting. This system would allow practitioners to explore risk perception, identify at-risk workers, and train people in a realistic setting without exposing them to physical risk.

For this study, we had four primary hypotheses to assess participants’ sense of Presence and the suitability of the MR platform for studying naturalistic risk-taking behavior of workers:

H1: The developed MR environment with passive haptics strengthens the workers’ self-report senses of Presence and senses of working on the roof of a two-story building.

H2: In the MR environment, workers’ reactionary physiological signals change under different experimental conditions (i.e., adding/ removing tactile augmentation of safety interventions) in accordance with Risk Compensation Theory, and provide objective measures of Presence.

H3: With heightened Presence, workers adjust their risk-taking behavior under different experimental conditions in accordance with Risk Compensation Theory.

H4: With heightened Presence, workers’ productivity changes under different experimental conditions in accordance with Risk Compensation Theory.

Detailed descriptions and methods for data collection and analysis appear in the following sections.

4 RESEARCH METHODS

Among all trades, residential roofers are disproportionately exposed to fall hazards when installing shingles [12, 13, 44]. Content analysis of incident reports from 1997-2018 showed that more than 15% of victims who fell from a roof were equipped with the required fall protection. The case scenario for this study was the installation of asphalt shingles on the pitched roof of a two-story residential building. To objectively monitor the risk-compensatory behavior of

HASANZADEH ET AL.: PRESENCE, MIXED REALITY, AND RISK-TAKING BEHAVIOR: A STUDY IN SAFETY INTERVENTIONS 2117

of death among roofers in the United States [12, 13]. To examine whether risk compensation plays a role in such fatality and injury rates, this study created an immersive, mixed-reality (MR) environment to provide subjects with risk-free interactions while assessing their risky behaviors under different levels of protection. Specifically, we used passive haptics to examine a worker’s spatial perception of completing a roofing task on a sloped roof, and we varied the provided safety interventions to change risk perception. In parallel, we investigated the role of such human factors as personality traits in determining subjects’ sense of Presence and engagement with the MR environment to confirm whether the environment effectively permits studying worker’s risk-taking behavior. Our methodology and validating results will foreseeably empower project managers to more actively mitigate incidents by providing a platform for identifying at-risk workers and generating personalized training.

2 BACKGROUND

2.1 Immersive Environment and Safety Training Using immersive environments to improve safety has gained popularity in many fields in recent years. Immersive environments (virtual reality-VR, augmented reality-AR, and mixed reality- MR) are used to simulate the hazardous situations in a risk-free setting, for example for acrophobia treatment [14] or emergency evacuation [15]. Therefore, the wide IE continuum serves as a new opportunity for improving the safety training process through interactive training methods that bring a real-life aspect into training scenarios.

CAVE virtual displays provide a relatively risk-free opportunity to re-enact on-the-job hazardous situations safely, experience real-time consequences, learn from mistakes, and improve the situational awareness of individuals. However, limited studies have used this technology for safety training as compared to more affordable techniques (e.g., a head-mounted display). In the following, the application of CAVE in military, mining, security, education, and construction domain are discussed.

The U.S. Army Research Laboratory developed a VR system consisting of a CAVE display and an Omni-Directional Treadmill to provide intensive training in hazardous military missions. The locomotion interface facilitates the coordination of the entire system, and as the trainee walks, runs, or jumps, the CAVE display changes appropriately [4]. Although no data were gathered to measure the effectiveness of this training, the authors took a huge step towards making CAVE a useful system for military training purposes. In another study, the five-sided CAVE was used to train detectives in an active shooter response scenario [16]. For security training, depicting a real scenario is very important. The deputy was equipped with the same equipment that they have on patrol, such as real weapons. The floor was designed to project the sound and simulate vibration of a terrifying high-school shooting. During the experiment, live-action 360-degree videos were played to assess the deputies’ responses and reactions. In some studies of CAVE-based VR environments, improving emergency responses have been examined. For example, Ericson (2007) also used a CAVE-based VR training environment to simulate fire situations to increase children’s understanding of fire safety, and to help children learn to escape in this hazardous situation for which practicing in real life is impossible [15]. In another study, a • ve-sided CAVE system was used to study individual’s risk perception and evacuation behavior during a threatening situation such as a tunnel emergency [17].

CAVE has been used as a safety training tool in construction equipment operation. In the study conducted by Yuen and his colleagues in 2010, forklift truck operation was modeled using CAVE-based VR with which drivers can practice forklift truck maneuvers and operations. To make the simulation as real as possible and enhance safety awareness of drivers, “virtual incidents” are created and visualized. The results of the study showed that operators become more vigilant and skillful in handling hazardous

conditions [18]. In another study within construction safety settings, Perlman et al. (2014) used two training techniques (photographs and construction documents as well as a virtual construction site) to investigate the differences in risk perception of superintendents. The results of their study showed that subjects in the virtual environment identified more hazards illustrating that the virtual environment portrayed hazardous situations better [2]. Taking into account the challenges and lessons learned from previous VR safety-related studies, in the present study, we have decided to use CAVE to create mixed-reality to monitor risk-taking behavior of workers.

2.2 Haptic Feedback and Human Spatial Perception Incorporating realistic haptic feedback in human-computer interaction is essential to immersive environments. There are two types of haptic interfaces: active and passive. Active haptics is computer-generated, while passive haptics provides feedback by simulating the shape, weight, or other physical properties of objects [19]. The use of passive haptics in combination with an active device could enrich interactions with virtual environments and significantly improve the user’s spatial perception and performance in a given task [20, 21].

One way to generate a sense of force/tactile feedback for the user in the IEs is to use active feedback hardware devices (such as force feedback gloves) [22]. However, the user needs to be attached to the device even when there is no force feedback, which makes the interaction unusual; and affects human-computer interaction in many ways; for example, the user’s sense of immersion is reduced significantly [21-24].

Contrary to active haptics, which relies on actuators driven by computer systems, passive haptics relies on real, physical objects or props that simulate the touch and force channels and enhance the user’s experience through the existence of a real physical object [21]. A disadvantage is that the passive haptic feedback cannot change during the simulation in real-time [25]. Martin and his colleagues recreated a mixed-reality to achieve realistic movements of a worker during completion of an assembly-like scenario, while movable props representing an object were adjusted to simulate the weight of the actual object [21]. The results of their study showed that simulating the form and weight of actual objects is the key to improve the VR experience of the participants. In one of the recent studies, Nagao and his colleagues instead of simulating a sense of vertical transportation using complicated systems or by using actuators, they have used passive haptic stimuli (i.e., small bumps under the feet of users) to simulate the edges of stairs in a virtual environment [26]. The results of their study showed that the small bumps as passive haptics enhanced the user’s feeling of Presence and the sense of ascending stairs.

Although the virtual-reality has been commonly used in the construction and civil engineering field, there are very limited studies that use immersive graphics and passive haptics to improve the safety learning. For example, for simulating the scaffolding system in a high-rise scenario, Shi and his colleagues used a wood plank in the real-world as a passive haptic combined with a virtual environment to provide workers with a greater tactile realism and sense of height [27]. Therefore, the previous literature made substantial efforts to show that passive haptics can improve the user’s spatial perception and experience in immersive virtual environments.

2.3 Presence in Immersive Environments Although IEs have been used for a wide variety of applications, one of the most important challenges is to determine the effectiveness of IEs, which has driven much research in this area [28-30]. The credibility of an experiment scenario in an immersive environment demands to model true-to-life events and user interactions [31, 32]. One of the common measures of IE effectiveness is the Presence concept, which was first established by Akin and his colleagues in 1983 [33]. Presence is a cognitive parameter and a subjective

phenomenon describable as the sensation of being in an MR environment [34]. They defined Presence as the feeling of “being there” in a virtual place. Witmer and Singer defined Presence as “a subjective experience of being in one place or environment, even when one is physically situated in another” [35].

Presence as a normal awareness phenomenon requires the direct attention of subjects. Involvement is encouraged, and immersion is enabled when there is a close interaction between an environmental factor and sensory stimulation. In fact, people experience various degrees of Presence when they are immersed in an IE, and this determines the extent to which they become involved in the situation portrayed [35]. The two orthogonal components of Presence contribute greatly to the realistic response of participants when they are immersed in a VR system [32]. Place Illusion (PI) means “being there” or “having a sensation of being in a real place” [32], p. 3549; and corresponds to the traditional conception of Presence. Plausibility Illusion (Psi) refers to the illusion that “the scenario being depicted is actually occurring” or believing in what is happening [32], p. 3549. PI is constrained by the virtual reality system properties in terms of its sensorimotor constraints, and Psi is impacted by the credibility of the situation as portrayed compared to expectations. It must be emphasized that, when PI breaks, because it is a perceptual illusion, it can be recovered very quickly, but when Psi breaks, it is unlikely to be recovered.

Previous literature determined that Presence is a multi-component construct affected by both technological factors (i.e., vividness) and human factors (e.g., subject personality, cognitive style, computer experience) [36-38]. In terms of technological factors, in the present study, if subjects experience the sensation of being on the roof of a two-story residential building and participate in the roofing scenario in an authentic way, the risk they perceive will resemble what they might experience when completing the same scenario in the real world. Thus, by combining passive haptics with a projection-based virtual environment, the mixed-reality triggers sensations of height as well as working and walking up and down a roof, despite the fact the subject only walks on a sloped surface built on floor level.

Since the Presence construct is subjective -a qualia (i.e., a subjective and internal feeling elicited by sense perceptions)- with no direct way to measure it, this construct has been measured in many different indirect ways in the literature [32]: self-reporting, physiological, and psychophysical/behavioral measures. All existing measures have advantages and disadvantages. The predominant method for measuring Presence is the use of post-trial self-reporting questionnaires. Skarbez and his colleagues reviewed the use of the most popular self-reporting questionnaires and found that these questionnaires are easy to apply because the questions are general and do not require substantial modification for various types of research [39].

Previous literature also showed that the use of physiological and behavioral responses as surrogates for Presence is methodologically and practically sound [40, 41]. Accordingly, if a person feels that s/he is in the roofing scenario depicted in the immersive mixed-reality environment, then s/he should exhibit changes in behavior and physiological responses concomitant with the fall-risk level. Therefore, this study was designed to elicit measurable physiological (i.e., heart-rate variability) and behavioral (i.e., changes in risk-taking behavior) responses under various experimental conditions.

2.4 Risk Compensation Theory The history of risk compensation (or risk homeostasis) dates back to two pioneering theories by Peltzman (1975) and Wilde (1982) [42], [9]. They argued that the benefits of a decreased risk of an injury associated with providing safety intervention are reduced significantly as a result of increased risk-taking behavior. Both theoretical backgrounds suggested that many safety technological advances and safety interventions may not be optimally effective due to this effect. Although the risk-compensation phenomenon has been studied in various disciplines, the previous studies mostly involved

participants who were never exposed to any life-threating decision-making during the experiment, and their risk-compensatory behaviors were assessed based on self-report ratings. For example, Feng and his colleagues used common self-reported ratings to imagined situations for examining risk-compensatory behavior in the construction industry [43]. Subjective measurement tools are not most appropriate for measuring risk-taking behavior in a high-risk industry such as construction, so, verbal responses to hypothetical scenarios cannot reflect the actual risk-taking behavior. While their study contributes to the body of knowledge, they could not find signs of compensatory behaviors. A proper study of the risk compensation effect requires to more objectively and empirically investigate the effect of safety interventions on individuals’ risk-taking behaviors. The present study suggested using a mixed-reality environment with passive haptics to capture naturalistic risk-taking behavior of workers and to study risk compensation effect in the construction industry. If the developed MR environment provides a higher sense of presence, and in turn, prompts individuals to adjust their risk-taking behavior in response to preventive interventions in accordance with the risk compensation theory; the proposed MR approaches can be considered as one of the promising methods to objectively study the workers’ risk-taking behavior affected by human factors.

3 POINT OF DEPARTURE

This study aimed to examine the undesirable latent side-effects of safety interventions, known as risk compensation. To address this objective, we changed the safety features in the experimental scenario within the mixed-reality environment to assess any associated changes in workers’ risk perceptions, risk-taking behaviors, and successive decision-making. In this paper, we evaluated whether mixed reality is a suitable platform for assessing risk perception and the risk-taking behaviors of construction workers. Specifically, we posited that if simulating height and virtual falls using passive haptics in a mixed-reality environment stimulate sufficient Presence to trigger subjects’ behavioral changes under different experimental conditions, then the MR platform will capture workers’ naturalistic risk compensatory behaviors within a risk-free setting. This system would allow practitioners to explore risk perception, identify at-risk workers, and train people in a realistic setting without exposing them to physical risk.

For this study, we had four primary hypotheses to assess participants’ sense of Presence and the suitability of the MR platform for studying naturalistic risk-taking behavior of workers:


H2: In the MR environment, workers’ reactionary physiological signals change under different experimental conditions (i.e., adding/ removing tactile augmentation of safety interventions) in accordance with Risk Compensation Theory, and provide objective measures of Presence.

H3: With heightened Presence, workers adjust their risk-taking behavior under different experimental conditions in accordance with Risk Compensation Theory.

H4: With heightened Presence, workers’ productivity changes under different experimental conditions in accordance with Risk Compensation Theory.

Detailed descriptions and methods for data collection and analysis appear in the following sections.

4 RESEARCH METHODS

Among all trades, residential roofers are disproportionately exposed to fall hazards when installing shingles [12, 13, 44]. Content analysis of incident reports from 1997-2018 showed that more than 15% of victims who fell from a roof were equipped with the required fall protection. The case scenario for this study was the installation of asphalt shingles on the pitched roof of a two-story residential building. To objectively monitor the risk-compensatory behavior of


4.4.1 Personality Measurement We administered 44-item BFI questionnaire to efficiently assess the five dimensions of personality (i.e., extraversion, agreeableness, conscientiousness, neuroticism, openness to experience) of workers. All participants completed the questionnaire and reported how accurately each trait describes themselves using a rating scale ranging from one (very inaccurate) to five (very accurate).

4.4.2 Presence Measurement

After training, the participant was asked to wear 3D goggles with head-tracking sensors, an ankle location-tracker bracelet on their left ankle, and Garmin Vívosmart® HR+ activity-tracking wristband. The participants then installed 27 asphalt shingles under three different safety-intervention conditions (Fig. 1). Under one condition, the subject completed the roofing task with only typical personal protective equipment (PPE) (e.g., hard hat, gloves, and knee pads) and no fall protection (a). Thus, the edge was unprotected and the likelihood of experiencing a virtual fall would be high. Under another condition (b), the participant completed the same task while provided with a fall-arrest system (Level-1 fall protection); consequently, the edge remained unprotected, but the likelihood of experiencing a virtual fall was lower due to the fall protection. Thirdly (c), the subject installed the shingles, while provided with a fall-arrest system and a guardrail protecting the edge (Level-2 fall protection).

To offset the order-effect in a repeated measures design with counterbalancing, the participants were randomly divided into three groups, each of which was assigned a different sequence of experimental conditions (I: abc, II: bac, and III: cba). Furthermore, since the intensity of the physical activity associated with the simulated roofing activity affects the participant’s heart rate—and to ensure his/her heart rate returns to normal between experimental conditions—the participants received 5–10-minute breaks between experiments.

The geometric center of the CAVE cubic room was labeled as (X,Y,Z) = (0,0,0), so the z-coordinate tracked workers’ entrances into risky areas and the onset of risky behaviors. When the worker remained in a risky zone, changes to his/her physiological responses were monitored via heart rate and heart-rate variability (HRV). HRV is an objective measure of experienced risk and is very sensitive to changes in the autonomic nervous system activity associated with stress, with higher HRV signifying a lower level of stress or arousal. HRV is computed based on inter-beat intervals and is equal to 60,000/Heart Rate beats per minute (bpm) and corrected for physiological artifacts. This metric indicated subjects’ physiological responses to Presence as well as their physiological response to their perceived level of risk. The risk-taking behavior of participants was also examined, based on their overall frequency (Eq. 1) and temporal exposure (Eq. 2) to fall risk, wherein we measured instances and durations of when the subject stepped into the risky areas as well as how they stabilized themselves in the risky zone. Lastly, under each experimental condition, the duration it took for each worker to complete the roofing task (installing 27 shingles) was considered as a measure of productivity. ��_��

�

��

��_��

��

��

(1) (2)

Where � represents the total interval of time used to complete

the roofing task, �� is a binary parameter explained above, �� represents minimum period for which the exposure time is calculated, �� represents instant value of the z-coordinates at the time t, and �� represents safety critical range of the z-coordinate [-0.3, 0].

After the experiment, the participant verbally answered the following question based on the single-item questionnaire by Bouchard et al. (2004) [48]: To which extent (0-5) do you feel present in the virtual environment as if you were really there? Explain? The conversation continued with follow-up questions: How did your risk perception change in each experimental condition? During which condition did you perceive a greater level of fall risk? Under which condition did you feel more protected? Then, the responses were coded by content analysis to examine whether participants’ changes in sense of Presence relate to their behaviors and whether participants who failed to perceive adequate risk were more likely to select risky behaviors. These results provided a robustness check for the overall results, as described below.

4.5 Evaluation Methods Different descriptive and inferential analyses were used to examine changes in a worker’s sense of Presence during the experiment to determine whether developing an MR platform in CAVE is one of the promising approaches for studying human factors in the construction safety setting. During the semi-structured interviews, the participants were asked about their experience with the mixed-reality: whether they experienced sickness, dizzyness, or any discomfort. As reported by participant, no participants stumbled on a roof due to dizziness or sickness or felt any discomfort, so we did not face any issues related to simulator sickness and safety. Furthermore, all 33 participants were able to complete the three experiments.

All box and whisker plots included here have the following features: the cross in each box is the mean, the solid line in the middle of the box is the median, the bottom and top of the box indicates the 1st quartile and 3rd quartile, and the T-bars extend to the 95% confidence interval. The horizontal lines with stars in the interaction graphs depict the significant pairs.

5 RESULTS AND DISCUSSIONS

5.1 Subjective Measure of Presence


The mixed-reality environment was designed to simulate a roofing task, so to determine whether we could capture the realistic response of participants to the scenario being depicted, the model required two orthogonal components discussed in literature: Place Illusion (PI) and Plausibility Illusion (Psi). PI means that the participant must feel they are on the roof of the two-story residential building, so as they turn their heads, they should see other buildings, the street, and other features of a suburban area. Such details will reinforce the feeling of PI. In turn, Psi means that the scenario being depicted is actually occurring, so we hoped adding gravity, sound, and wind effects to the mixed-reality model would reinforce the feeling of Psi. Figure 2 shows the heights of the roof in the real and mixed-reality environment.

Fig. 2. Schematic of experimental environment versus simulated sensation. Presence was produced by the slopped roof and the visualization of a two-story height in the mixed-reality model.

workers, the hazardous situation must be simulated in such a way as to give the impression that the subject is at-risk. According to Institutional Review Board, we would not be allowed to conduct this study on real-world construction site and to expose workers to high-risk scenarios without providing them with appropriate protective equipment. Therefore, we established an immersive mixed-reality environment to simulate the roofing task, monitor the risk-taking behavior of workers, and examine whether risk compensation offsets the benefits of safety interventions designed to prevent injury from falls.

4.1 Apparatus Participants were put on a virtual second story roof and asked to install several courses of real asphalt shingling on a real sloped roof. This experiment used a CAVE Automatic Virtual Environment (10 feet cubed) to create an immersive MR environment with a high-resolution, stereoscopic, and head-coupled panoramic views. Our CAVE is a projection-based immersive virtual reality system with eight active stereo projectors blended across four display surfaces to make 2560 pixels square on each side of the 3 back-projected walls and the top-projected floor. The first-person, stereoscopic perspective view is rendered via optical head-tracking. We chose the CAVE environment for its large, close-to-human field of view as well as for its capacity for subjects to perceive their own body and local objects directly [34, 45]. We believe the CAVE system facilitates a high sense of Presence since subjects can see the physical as well as virtual world, which is projected (in perspective) around them. The CAVE’s ability to add multiple depth cues and passive haptics assured us that the environment was real enough to suspend the subject’s disbelief for a period of time [45]. Thus, CAVE was the ideal apparatus to study Risk-Compensation with a mixed-reality model.

4.2 Mixed-Reality Modeling In our mixed-reality environment, virtual and physical objects co-exist and interact in real-time, which requires co-development and synchronization of several components. Here, our virtual model depicts a suburban area in the U.S. consisting of one/two-story residential buildings. We built the 3D model in Maya and then rendered the scene at real scale using the ISO-IEC 19774 standard Extensible 3D (X3D) [46, 47]. The X3D environment was projected into our CAVE apparatus in real-time, rendered according to the first-person user’s position and orientation (perspective). The virtual model also provided such cues as traffic signs and naturalistic lighting to make the setting more visually realistic and convincing.

The physical element of our test environment used a real prop —a 2.4m-by-1.8m physical section of the roof with a 27-degree slope—to reproduce a real environment’s physical characteristics and to simulate the touch and force channels. As our participants were to complete a roofing task in a (simulated) hazardous environment, a goggle-based VR environment (head-mounted display, HMD) would not provide the same sense of Presence as a physical sloped roof, a hammer and shingles, as well as the visual and somatic feedback experienced during the experimental task.

Thus, building our mixed reality environment required five integrated systems in order to test whether we could stimulate naturalistic behaviors in a virtual, risk-free space:

1.VR tracking tools: Two 6 DOF tracking targets (the head-target mounted on the shutter glasses and ‘wrist’ target affixed to the ankle) were used to collect the physical pose of each individual and to register interactive behaviors (e.g., adjusting the view or to initiate a virtual fall animation):

i. Users were presented a scene with strong depth cues (head-coupled stereoscopy, occlusion, motion parallax, and relative distance).

ii. Users could lean up to the edge of the roof to see the backyard two stories below; going too far triggered a ’fall’ animation, dropping the perspective to the ground.

2. Synchronize the virtual and passive haptics: The key to deploying mixed-reality systems is to confirm that the virtual world and the physical world remain registered and synchronized at all times. In order to evoke a realistic visuo-haptic interaction, the passive haptic’s physical object must align with the virtual model. Thus, we modified the projection parameters for one floor projector to match the slope of the physical prop. For this experiment, the sloped roof was positioned with its high side at the open back of the three-walled CAVE. Thus, we specified the new ‘screen’ geometry as the physical roof prop, rather than the typical flat floor.

3. Log user behaviors: Thresholds were defined to code subjects’ head and ankle locations continuously—for example:

i. When the subject moved within 0.6 meters of the roof edge (0<z<0.3m), the data point was tagged as a “near-miss” fall.

ii. When the subject moved within 0.3 meter of the roof edge (-0.3m<z<0), the likelihood of a fall increases, so the algorithm tags the data point as “risky.”

iii. If the subject stands on an edge and leans over the roof, the MR algorithm initiates a virtual ‘fatal’ fall.

4. After setting up the scenario and relevant behaviors, the dynamic behaviors of the mixed-reality environment were tested and validated. Three conditions were tested:

i. No fall protection (Fig. 1a) ii. Level-1 fall protection: fall arrest system consists of a body

harness, anchorage, and connector (Fig. 1b). This injury-reducing intervention has a higher risk associated with it because when erected a fall can occur but the fall is arrested within acceptable force and clearance margins.

iii. Level-2 fall protection: fall arrest system and guardrail (Fig. 1c).

5. Add environmental effects: To improve the mixed-reality environment, we focused on simulating an actual job site at a suburban area with other modalities:

i. a wind effect was simulated—using two static fans with constant speed—

ii. a sound effect played common suburban ambient noise for 9:00 AM (e.g., birds are singing, the car is passing, children are playing).

4.3 Participants To test the Risk-Compensation effect related to protective equipment independent of biases associated with experience-related factors, students were recruited to complete the roofing task in the mixed-reality environment. Thirty-three healthy participants (28 male and five female) aged 23.3 ± ~2.3 years participated in the study. All subjects had at least one year’s experience in the construction industry, and all had assisted supervisors and other trades in the completion of construction tasks at job sites. All research procedures were approved by the Institutional Review Board (IRB), and participants received gift cards as compensation for participating.

4.4 Experimental Design and Data Collection We used a within-subjects experimental design wherein each subject’s test was conducted in two hours in a single day. Before initiating our experiments, all participants signed an informed consent form and received explanations regarding the study. Then, a full training session (20-min) was given to participants regarding how to complete the designated roofing task while remaining safe on the roof. The participants also were asked to fill out several surveys (e.g., demographic, personality, sensation-seeking, risk tolerance, and locus of control) and to complete a computerized risk-taking behavior assessment game (i.e., Balloon Analogue Risk Task/BART game). In this paper, we discuss the results of the big-five personality traits test (BFI) as a robustness check for the participant’s sense of Presence during the experiment, as found below.


felt under conditions b and c—participants perceived a lower risk of falling when the edge was protected by a guardrail (condition c) compared to when it is unprotected (conditions a and b). In sum, the realism of mixed-reality design and experimental conditions apparently made a significant difference in workers’ risk perception. Figure 4a demonstrates the distributions of all participants’ physiological signals (i.e., heart-rate variability) under each experimental condition.

5.3 Psychophysical/Behavioral Responses as a Surrogate Measure of Presence

5.3.1 H3: With heightened Presence, workers adjust their risk-taking behavior under different experimental conditions in accordance with Risk Compensation Theory.

In contexts of behavioral changes, Presence is measured by non-intrusively monitoring the behavior of participants in response to different stimuli to examine the extent to which participants experienced PI and Psi in the MR environment. These stimuli are meant to trigger specific behaviors such as anxiety/stress because of a fear of heights and of a fall, or precautionary vs. increased risk-taking behavior because of safety interventions. The specific stimuli and the methods for monitoring the behavior therefore become very important to ensure that we capture the participants’ natural behaviors. Specifically, if the changes in risk-taking behavior of participants resemble what we expect to see at a job site (based on Risk Compensation Theory), these behavioral changes can demonstrate the participants’ sense of Presence. Although an increased risk-taking behavior does not necessarily result in an injury or a fatality, such at-risk behavior increases the likelihood of being injured, so the risk-taking behavior of workers was evaluated based on the three metrics: fall-exposure time, frequency of at-risk behavior, and changes in stabilizing behaviors.

(a) (b) Fig. 4. (a) Physiological responses (HRV) and (b) Fall exposure duration of all participants under three experimental conditions

To understand the differences in participants’ risk taking via fall-exposure durations under the three experimental conditions (i.e., a, b, c), we conducted a repeated measure ANOVA. Mauchly's test indicated that the assumption of Sphericity had not been violated,

2(2) = 1.625, p = 0.44> 0.05. A repeated-measures ANOVA identified a statistically significant mean fall exposure duration between the experimental conditions (F(2, 64) = 8.259, p = 0.01< 0.05). Post hoc tests using the Bonferroni correction revealed that increasing the level of safety intervention elicited a slight increase in fall-exposure time from condition a to b (2.38 ± 0.59 min vs 2.62 ± 0.74 min, respectively) and from condition b to c (2.62 ± 0.74 min vs 2.92 ± 0.88 min, respectively), which were not statistically significant. However, participants exposed themselves to fall risks over longer periods when provided with the highest level of protection (condition c), which was statistically significant as compared to condition a (p= 0.03< 0.05). Figure 4b demonstrates the distribution of the total duration (i.e., minutes) all subjects spent in risky zone under each condition. In sum, while the participants were

aware of their fall risk, they generally spent more time exposed to fall risks after adding the tactile augmentation of safety interventions (conditions b and c). Consequently, our hypothesis that their level of Presence would correlate with risk-taking behaviors appear valid in terms of risk-exposure durations.

To deepen the discussion here, the frequency at which participants entered the risky zone was also monitored. The heatmaps in Figure 5 depict the frequency at which subjects stepped into the risky areas when completing the roofing task under each experimental condition. Participants dared to step into risky areas more frequently after adding tactile augmentation of safety interventions (condition b and c). This outcome concurs with the risk-compensation behaviors observed in [10], which further illustrates the validity of our H3.

Fig. 5. Heatmaps demonstrating all subjects’ risk-taking behaviors in terms of the frequency of proximity to risky zone (0<z<0.3m) under each experimental condition.

Head and ankle coordinates were used to locate participants in the environment to study how they stabilized themselves on the roof under each experimental condition. Roofers installing three-tab asphalt shingles usually face uphill; however, the postural analysis in this study revealed that our participants changed their facing directions under different experimental conditions. They might stabilize themselves by facing uphill or by facing sideways based on their perceived level of fall risk in each condition. For each experimental condition, when the participants stepped into the risky zone, the frequency (percentage) at which they stabilized themselves using different facing directions was calculated. Figure 6 demonstrates the probability density estimations of all participants facing directions (uphill or sideways) under each experimental condition.

Fig. 6. Workers’ facing directions under each experimental condition

In terms of Presence, these behavioral changes become an interesting consideration. Under condition a, where the participants had no fall protection and the edge of the roof was unprotected, participants took precautionary actions by stabilizing themselves on the roof while facing down. During interviews, participants justified

The post-trial Presence measure was administered during the semi-structured interview session. Semi-structured interviews continued to elaborate more on the subjects’ sense of Presence in the designed MR. Regarding the sense of height, the results of Presence question showed that 88% of participants believed that the MR simulation was realistic enough to simulate the height of a two-story residential building. Figure 3 shows the Presence scores (red line) reported by each participant. Regarding the sense of Presence, the self-report Presence scores of the mixed-reality model were very high (Mean= 4, SD= 0.8). The high scores demonstrated that passive haptic feedback corresponding to the sloped roof enhanced the participant’s sense of Presence.

Participants reported many features embedded in our MR strengthened their sense of working on the roof of a two-story building: the sound and wind effects; the realistic virtual projection of a suburban environment; and especially the slope, which destabilized their posture and made them take care not to fall. They underscored that looking down what appeared to be the edge of the roof created a sense that they were two stories high, despite the fact that they were actually working on a sloped physical model on the floor (Fig 2).

Therefore, the subjective description of Presence confirmed that building a simple passive haptic in the position corresponding to the slope and edge of the roof in the virtual environment was an effective method for capturing the naturalistic risk-taking behavior of workers. However, we can argue that the self-report Presence questionnaires are post-trial, so they present an intrusive method and may not represent the natural feeling of individuals regarding what they have experienced during the experiment. Moreover, the inherent bias associated with subjective measurement tools makes the outcome of these methods challenging to interpret.

5.2 Physiological Reaction as a Surrogate Measure of Presence

H2: In the MR, workers’ reactionary physiological signals change under different experimental conditions in accordance with Risk Compensation Theory, and provide objective measures of Presence.

Due to limitations associated with the self-reporting measures, objective measures (i.e., behavioral, physiological, and psychophysical measures) are suggested for measuring Presence. Beneficially, physiological measures (e.g., fMRI, EEG, heart rate, skin response) can be used to assess many of the shortcomings of self-reporting measures by gathering continuous and real-time data. Among the factors presenting in physiological measures, perception of risk is a core physiological “stressors” (Kim et al. 2018), so measuring stress metrics can signify subjects’ sense of risk. Concurrently, a typical physiological measure for Presence relates to the physiological characteristics of arousal/stress, as measured by increasing heart rate and decreasing heart-rate variability.

Our study used a Garmin Vívosmart® HR+ activity tracker to monitor these physiological responses among our participants while they completed the roofing task under each experimental condition. We anticipated that if the mixed-reality environment sufficiently

resembled a real-world construction site, it would evoke physiological responses similar to those evoked by the corresponding real construction environment. Thus, our hypothesis expected that changes in the experimental condition in the MR environment would also change physiological responses to the environment.

Obviously the physical activity of the simulated roofing task would reasonably raise heart rates compared to rested heart-rate as a baseline. While heart rate measures the average beats per minute, HRV focus the specific changes in time between successive heartbeats and has been accepted as the most precise non-invasive measurement of stress and mental load [49]. We then calculated heart-rate variability, which provided us with information on the state of the participant’s autonomic nervous system and each subject’s internal/external psychological stressors. On average, participants reported high sense of Presence (Mean= 4, SD= 0.8) in the mixed-reality passive haptic, and they reported subjectively that their perceived risk changed under various experimental conditions. This is a sign that the subjects engaged in the roofing task as they would at an actual job site with fall-risk. The non-significant correlational analysis showed that generally, 19% variations in physiological responses correspond to participants’ sense of Presence. Figure 3 also demonstrates that among those participants who reported higher subjective Presence score (Presence score> 3), generally, more variation manifested in their physiological responses under the various experimental conditions. However, further investigation is needed to explore whether changes in mixed-reality safety features significantly project in the physiological stress responses of participants as a measure of perceived risk.

As discussed before, in this paper, the worker’s risk-taking behavior was studied in the context of Risk Compensation Theory. According to this theory, in hazardous situations, workers’ behaviors are induced based on the risks (i.e., perceived risk of injuries or fatalities) and benefits (positive desired outcomes, such as excitement or timesaving) associated with the situation [10]. Therefore, if the physiological responses followed the expected results under Risk Compensation Theory (i.e., as the mixed-reality safety features increase, the perceived risk and stress level will decrease), we could conclude a higher sense of Presence among participants.

A Kruskal-Wallis H test was conducted to examine whether the heart-rate variability (as a measure of stress or perceived risk) changes over different experimental conditions. The results show that workers experienced a significantly lower level of stress in the risky zone when provided with higher levels of safety interventions ( ² = 11.31, p= 0.00< 0.05). Pairwise comparisons revealed statistically signi• cant differences in heart-rate variability between high-safety intervention condition c (mean rank= 63.45) and no-intervention condition a (mean rank= 40.91) (Adj. p= 0.00< 0.05), and between condition c (mean rank= 63.45) and moderate-safety intervention condition b (mean rank= 45.64) (Adj. p= 0.03< 0.05).

During semi-structured interviews, the participants’ explanations for the changes in their perceived level of risk provided insights into the physiological measure of Presence. They expressed their safety concerns in condition a and the increased sense of protection they

Fig. 3. Bar plots demonstrate heart-rate variability under each experimental condition and subjective measure of Presence for each participant. The left-vertical axis represents subjective Presence score and the right-vertical axis represents HRV.

350400450500550600650700750800

00.51

1.52

2.53

3.54

4.55

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

Presence Score

ParticipantsHRV_a HRV_b HRV_c Presence Score


the sensation of height. This finding aligns well with previous studies, which suggested that highly open individuals actively sought information in the surrounding environment and processed the cues in more in-depth manner [57], and argued that this deeper processing of the scene indicates that individuals who scored higher in openness experience stronger feelings of Presence [58].

Although heterogeneous results were discussed in the literature in terms of the relationship between big-five personality traits and Presence, this study showed that openness and extraversion correlate with Presence. We suspect these findings stem from the simulated roofing task. The studied personality traits have proven to correlate with an individual’s risk-taking behavior—extroverts and individuals who are open to experience are categorized as risk-takers [56]. Therefore, since the simulated mixed-reality environment in this study involves a high-risk scenario, Presence with this environment would likely coincide with those who are extroverts or more open to experience higher level of engagement. To deepen the discussion here, the participants were grouped based on their extroversion and openness scores (above and below mean). The boxplots in Figure 8 reveal the most extroverted workers tended to feel more present in the MR environment than the introverts and the higher scores in openness enhanced the sensation of Presence.

Fig. 8. Interaction between personality traits (i.e., extraversion and openness) and objective Presence measures

6 LIMITATIONS AND RECOMMENDATIONS

We are cautious about generalizing our results because some limitations exist related to this research that can be addressed by future studies. First, our relatively small sample size limits the generalizability of the present study, so future studies should replicate this study with an increased number of participants to address this limitation. Second, the present study was conducted only for individuals, so only one participant was present in a scenario, had control over the MR system, and controlled the viewpoint based on his/her head position. Future studies are recommended to examine the effect of peers’ Presence in producing the sense of being there or to allow several participants to interactively control the system and switch the viewpoint. Third, the current study considered only two personality traits and examined each of them separately to study their impact on Presence measures. Personalized immersive environments can enhance safety learnings and complement conventional safety

training, so, future studies are encouraged to investigate the possible combination of personality characteristics and the demographic factors that may impact subjects’ sense of Presence.

7 CONCLUSION

We performed a controlled experiment investigating the potential of using mixed-reality with passive haptics to monitor workers’ risk-taking behaviors and to examine the risk-compensation effect in the construction industry. We adopted multi-model techniques by synchronizing the virtual environment with passive haptics to produce a mixed-reality system that captures the naturalistic risk-taking behaviors of workers and simulates virtual fall hazards. In our mixed-reality, we established a passive haptic roof section within a virtual suburban environment, and provided three different conditions of safety interventions. The mixed-reality environment provided a natural scale, depth (simulating human stereoscopy), texture, and shadow to facilitate a sense of Presence and height.

The findings demonstrated the expected direct relationship between the increasing level of protection and the reduced perceived level of risk and increased risk-taking behavior of workers. Thus, the environment triggered workers’ sense of Presence in accord with Risk Compensation Theory. Our objective data monitors showed that while the participants had sure knowledge of their mixed-reality environment, the well-synchronized virtual environment and passive haptics caused the participant to perceive the simulated roofing task in the mixed-reality world similar to the real-world. Thus, the MR provides a unique opportunity to raise the participants’ sense of Presence and capture their realistic response to safety features for both research and training purposes. Our robustness checks also make this study one of the first studies to use an immersive mixed-reality system to investigate the relationship between personality characteristics and the subjective and objective measures of Presence in the construction safety setting. Technological advances used in this study helped certain personality traits become immersed in the mixed-reality environment and led to a greater sense of Presence. Specifically, the results of this study showed that workers who are extroverts or open to new experience could place themselves more readily within this mental representation and experience higher levels of involvement, higher degrees of spatial Presence, and a stronger sense of being there.

As noted before, there are some limitations exist in the present study which limit the generalizability of the findings, and future study can address the limitation and broaden the contribution to the body of knowledge by replicating the study with an increased number of participants who interactively control the system. Despite these limitations, this paper contributes to the body of knowledge by describing a simulated mixed-reality environment with passive haptics that enabled researchers—and conceivably safety managers—to safely interrogate the risk-compensation effect as manifested in workers’ naturalistic responses. These results suggest that by applying passive haptics to our mixed-reality systems, we are able to produce sensations closer to those of the real world. Furthermore, the combination of different subjective and objective measures to assess the sense of Presence in the MR seems to be an optimal way to study the risk-taking behavior of workers and to expand future research into human-computer interactions. Since the results showed that risk-compensatory behaviors of roofers counteract the safety benefits of safety interventions, in the next step, we aim to use the findings of the present study and design a VR evidence-based intervention tool to raise awareness of the risk-compensation phenomenon among safety professionals and specifically among roofers in the construction communities.

ACKNOWLEDGMENTS The authors wish to thank VT Advanced Research Computing the Visionarium Lab for the resources and support for this study, and Drs. Joseph L. Gabbard, Lance Arsenault, Michael Garvin, and E. Scott Geller for their considerable and vital support in this study.

this choice as providing more control over their bodily situation—facing sideways is unfavorable in practice as it may yield lower back disorders. On the other hand, when participants possessed fall-protection systems and when the edge was protected by a guardrail, the roofers reported feeling more secure and took more risks by stepping too close to the edge, leaning on the guardrail’s toe-board, and facing uphill while installing shingles. Hence, the changes in safety features provided in the mixed-reality environment led to different psychophysical responses, which in turn manifest a level of Presence. However, we could not find any significant difference in facing directions among the experimental conditions.

5.3.2 H4: With heightened Presence, workers’ productivity changes under different experimental conditions in accordance with Risk Compensation Theory.

To determine whether changes in completion time varied as a function of the experimental conditions (i.e., within-subject factor), we conducted a mixed-design ANOVA. The results show that there was a statistically significant difference in mean completion under the different experimental conditions; F(2, 60) = 5.59, p= 0.00 <0.05. Post hoc tests with a Bonferroni adjustment were conducted to determine which of these experimental conditions differ from each other. The post hoc analysis revealed that subjects’ productivity in terms of completing the task faster was statistically significant when increasing from no fall protection (a) to Level-2 fall protection (c) (p= 0.04< 0.05), and from Level-1 fall protection (b) to Level-2 fall protection (c) (p= 0.00< 0.05), but not from no fall protection (a) to Level-1 fall protection (b) (p= 1.00> 0.05).

A slight increase in completion time was seen when the participants used the fall-arrest system (see blue bars in Fig 7); this increase resulted from the frequent need to move the lanyard out the way during installation and translates into a loss of productivity for many residential roofing contractors. However, workers in our test appeared to compensate for this productivity reduction by enacting more risky behaviors—i.e., over-relying on their lanyard as a handhold while moving up and down the roof surface and expending more time in risky zone. Furthermore, our findings revealed that adding the guardrail passive haptic encouraged participants to complete the task faster (see blue bars in Fig 7). However, this increased productivity coincided with exposed fall-risk durations (see pink bars in Fig 7).

Fig. 7. Variances in productivity among experimental conditions: Total completion time and fall-exposure duration for all participants.

The red arrows illustrate the percentages of the time that participants exposed themselves to fall risk while completing the task under each condition. For instance, under condition a, on average, participants spent 13% of total time for completing the task in risky areas; and under condition c, on average, participants spent 17% of total time for completing the task in risky areas. Therefore, workers focused on productivity (in terms of saving time) and disregarded safety precautions that would slow them down or otherwise interfere with earning the immediate productivity benefit they perceived they gained from the safety interventions. The mixed-reality developed for this study could help to examine how the safety

interventions might become counterproductive because of the risk-compensation effect.

5.4 Robustness Checks: Effect of User’s Characteristics such as Personality Traits on the Sense of Presence

While the subjects’ characteristics have been discussed in the literature as influential factors to moderate Presence, they have not been adequately researched to date. Consequently, the relationship between the individual characteristics (i.e., personality traits) and their appreciation of the “realism” or “sense of being there” in MR environments presents an interesting less-explored topic. Therefore, we conducted correlation analyses to examine whether different personality traits (extroversion and openness) relate to the subjective (i.e., Presence survey) and objective (body movement, physiological responses, and risk-taking behavior) Presence measures in MR. Since the normality assumptions were violated, non-parametric Kendall-tau correlational analysis was used here. Although both extroversion and openness are slightly positively correlated with subjective (self-report) Presence measures, no significant correlations were found between them (rextraversion=0.24, pextraversion=0.08> 0.05; ropennes=0.19, popennes=0.19> 0.05) (Table 1). Table 1. Correlations between the different Presence measures and the selected Big Five personality traits

Personality Traits

Subj. Score

Exp a

Exp b

Exp c

HRV a

HRV b

HRV c

Extraversion 0.24 0.15 0.11 0.18 0.27* 0.35* 0.26* Openness 0.19 0.23 0.36* 0.15 0.15 0.26* 0.13 *� < 0.05

As discussed before, since this research was founded on workers’ risk-compensatory behaviors, if the participants changed their behaviors in meaningful ways due to their experience in the MR environment, the changed behaviors signified their Presence. The results showed a significant positive correlation between extroversion and the physiological responses depicting Presence (rHRV_a=0.27, pHRV_a=0.03< 0.05; rHRV_b=0.35, pHRV_b=0.00< 0.05; rHRV_c=0.26, pHRV_c=0.04< 0.05). Positive correlations also indicated that extroverted workers experience lower levels of stress and arousal (i.e., higher heart-rate variability) under each experimental condition.

All conditions combined, extroverted workers experienced a significant reduction in their level of stress and perceived fall risk when they received higher levels of safety interventions. Furthermore, previous studies argued that individuals who are highly extroverted tend to be more adventurous and more likely to take risks [50, 51]. Our results saw positive correlations between extroverts and risk-taking behaviors as well as present a stronger feeling of Presence in the MR. Thus, our finding aligns well with a study conducted by Laarni and his colleagues (2004), which argued that extroverts have more processing resources, perceive more information per time, and are more likely to feel present in immersive environments [52].

This study also found positive correlations between openness and changes in physiological and psychophysical responses of participants under each experimental condition. We can conclude that more openness to experience is associated with experiencing higher levels of Presence within the test environment. An explanation behind this positive relation is that individuals who score high in openness to experience are identified as curious, imaginative, and intellectual [36, 53, 54] and willing to take risks [55, 56]. It must be noted that the correlations are the highest (and significant at a critical value of 0.05) under experimental condition b (rExp_b=0.36, pExp_b=0.00< 0.05; rHRV_b=0.26, pHRV_b=0.05). The reason for these high correlations may be that in this experiment, the participants were equipped with the fall-arrest system but without a guardrail; and adventurous workers (who have higher openness scores) stepped into risky areas and spent longer periods there, conceivably to experience

13% 15% 17%


[48] S. Bouchard, J. St Jacques, G. Robillard, and P. Renaud, "A hint on the relationship between anxiety and presence," Presentation at Cybertherapy 2004 Conference, 2004.

[49] H. Jebelli, B. Choi, and S. Lee, “Application of wearable biosensors to construction sites. I: Assessing workers’ stress,” Journal of Construction Engineering and Management, vol. 145, no. 12, pp. 04019079, 2019.

[50] V. Golimbet, M. Alfimova, I. Gritsenko, and R. Ebstein, “Relationship between dopamine system genes and extraversion and novelty seeking,” Neuroscience and behavioral physiology, vol. 37, no. 6, pp. 601-606, 2007.

[51] A. Sacau, J. Laarni, and T. Hartmann, “Influence of individual factors on presence,” Computers in Human Behavior, vol. 24, no. 5, pp. 2255-2273, 2008.

[52] S. Böcking, A. Gysbers, W. Wirth, C. Klimmt, T. Hartmann, H. Schramm, J. Laarni, A. Sacau, and P. Vorderer, "Theoretical and empirical support for distinctions between components and conditions of spatial presence," Proc. of the Seventh Annual International Workshop on Presence, pp. 224-231, 2004.

[53] G. Saucier, “Mini-Markers: A brief version of Goldberg's unipolar Big-Five markers,” Journal of personality assessment, vol. 63, no. 3, pp. 506-516, 1994.

[54] O. P. John and S. Srivastava, “The Big Five trait taxonomy: History, measurement, and theoretical perspectives,” Handbook of personality: Theory and research, vol. 2, no. 1999, pp. 102-138, 1999.

[55] J. F. Salgado, “Big Five personality dimensions and job performance in army and civil occupations: A European perspective,” Human Performance, vol. 11, pp. 271-288, 1998.

[56] S. Hasanzadeh, B. Dao, B. Esmaeili, and M. D. Dodd, " Role of personality in construction safety: investigating the relationships between personality, attentional failure, and hazard identification under fall-hazard conditions," Journal of construction engineering and management, vol. 145, no. 9, pp. 04019052, 2019.

[57] J. F. Rauthmann, C. T. Seubert, P. Sachse, and M. R. Furtner, “Eyes as windows to the soul: Gazing behavior is related to personality,” Journal of Research in Personality, vol. 46, no. 2, pp. 147-156, 2012.

[58] S. E. Kober, and C. Neuper, “Personality and presence in virtual reality: Does their relationship depend on the used presence measure?,” International Journal of Human-Computer Interaction, vol. 29, no. 1, pp. 13-25, 2013.

REFERENCES [1] H. Li, G. Chan and M. Skitmore, “Visualizing safety assessment by

integrating the use of game technology,” Automation in construction, vol. 22, pp. 498-505, 2012.

[2] A. Perlman, R. Sacks, and R. Barak, “Hazard recognition and risk perception in construction,” Safety science, vol. 64, pp. 22-31, 2014.

[3] H. Hsiao, and P. Simeonov, “Preventing falls from roofs: a critical review,” Ergonomics, vol. 44, no. 5, pp. 537-561, 2001.

[4] H. P. Crowell III, J. A. Faughn, P. K. Tran, and P. W. Wiley, "Improvements in the Omni-Directional Treadmill: Summary report and recommendations for future development," ARMY RESEARCH LAB ABERDEEN PROVING GROUND MD, 2006.

[5] K. J. Stroud, D. L. Harm, and D. M. Klaus, “Preflight virtual reality training as a countermeasure for space motion sickness and disorientation,” Aviation, space, and environmental medicine, vol. 76, no. 4, pp. 352-356, 2005.

[6] S. Gupta, D. Anand, J. Brough, M. Schwartz, and R. Kavetsky, "Training in Virtual Environments: A Safe, Cost Effective, and Engaging Approach to Training," CALCE EPSC Press, University of Maryland, College Park, MD, 2008.

[7] M. Kassem, L. Benomran, and J. Teizer, “Virtual environments for safety learning in construction and engineering: seeking evidence and identifying gaps for future research,” Visualization in Engineering, vol. 5, no. 1, pp. 16, 2017.

[8] F. Bosché, M. Abdel-Wahab, and L. Carozza, “Towards a mixed reality system for construction trade training,” Journal of Computing in Civil Engineering, vol. 30, no. 2, pp. 04015016, 2015.

[9] G. J. Wilde, “The theory of risk homeostasis: implications for safety and health,” Risk analysis, vol. 2, no. 4, pp. 209-225, 1982.

[10] S. Hasanzadeh, and J. M. de la Garza, "Understanding Roofer’s Risk Compensatory Behavior through Passive Haptics Mixed-Reality System," Computing in Civil Engineering 2019: Visualization, Information Modeling, and Simulation, pp. 137-145, 2019.

[11] S. Hasanzadeh, J. M. de la Garza, and E. S. Geller, “Latent Side-Effects of Safety Interventions through Passive Haptics Mixed-Reality System,” Journal of Construction Engineering and Management, submitted for publication.

[12] Y. Kang, “Use of Fall Protection in the US Construction Industry,” Journal of Management in Engineering, vol. 34, no. 6, pp. 04018045, 2018.

[13] Y. Kang, S. Siddiqui, S. J. Suk, S. Chi, and C. Kim, “Trends of fall accidents in the US construction industry,” Journal of Construction Engineering and Management, vol. 143, no. 8, pp. 04017043, 2017.

[14] D. Opdyke, J. S. Williford, and M. North, “Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia,” Am J psychiatry, vol. 1, no. 152, pp. 626-28, 1995.

[15] E. R. Ericson, “Development of an immersive game-based virtual reality training program to teach fire safety skills to children,” 2007.

[16] C. Fink, "VR training next generation of workers," https://www.forbes.com/sites/charliefink/2017/10/30/vr-training-next-generation-of-workers/#a2be5f064f51. 2017.

[17] M. T. Kinateder, H. Omori, and E. D. Kuligowski, The use of elevators for evacuation in fire emergencies in international buildings. US Department of Commerce, National Institute of Standards and Technology, 2014.

[18] K. Yuen, S. Choi, and X. Yang, “A full-immersive CAVE-based VR simulation system of forklift truck operations for safety training,” Computer-Aided Design and Applications, vol. 7, no. 2, pp. 235-245, 2010.

[19] M. Gutierrez, F. Vexo, and D. Thalmann, Stepping into virtual reality. Springer Science & Business Media, 2008.

[20] H. G. Hoffman, A. Hollander, K. Schroder, S. Rousseau, and T. Furness, “Physically touching and tasting virtual objects enhances the realism of virtual experiences,” Virtual Reality, vol. 3, no. 4, pp. 226-234, 1998.

[21] N. A. A. Martin, V. Mittelstädt, M. Prieur, R. Stark, and T. Bär, “Passive haptic feedback for manual assembly simulation,” Procedia CIRP, vol. 7, pp. 509-514, 2013.

[22] S. Garbaya and U. Zaldivar-Colado, “The affect of contact force sensations on user performance in virtual assembly tasks,” Virtual Reality, vol. 11, no. 4, pp. 287-299, 2007.

[23] D. Clark, and M. J. Bailey, "Virtual-virtual haptic feedback and why it wasn't enough," Visualization and Data Analysis, pp. 308-318, 2002.

[24] G. Ye, J. J. Corso, G. D. Hager, and A. M. Okamura, "Vishap: Augmented reality combining haptics and vision," Proc. International Conference on Systems, Man and Cybernetics, pp. 3425-3431, 2003.

[25] R. D. Joyce and S. Robinson, "Passive haptics to enhance virtual reality simulations," Proc. AIAA Modeling and Simulation Technologies Conference, pp. 1313, 2017.

[26] R. Nagao, K. Matsumoto, T. Narumi, T. Tanikawa, and M. Hirose, “Ascending and descending in virtual reality: Simple and safe system using passive haptics,” IEEE transactions on visualization and computer graphics, vol. 24, no. 4, pp. 1584-1593, 2018.

[27] Y. Shi, J. Du, C. R. Ahn, and E. Ragan, “Impact assessment of reinforced learning methods on construction workers' fall risk behavior using virtual reality,” Automation in Construction, vol. 104, pp. 197-214, 2019.

[28] J. Blascovich, J. Loomis, A. C. Beall, K. R. Swinth, C. L. Hoyt, and J. N. Bailenson, “Immersive virtual environment technology as a methodological tool for social psychology,” Psychological Inquiry, vol. 13, no. 2, pp. 103-124, 2002.

[29] R. J. Lamson, "Virtual reality immersion therapy for treating psychological, psychiatric, medical, educational and self-help problems," Google Patents, 2002.

[30] B. E. Insko, M. Meehan, M. Whitton, and F. Brooks, “Passive haptics significantly enhances virtual environments,” Dept. of Computer Science, University of North Carolina at Chapel Hill, NC, 2001.

[31] C. Heeter, “Being there: The subjective experience of presence,” Presence: Teleoperators & Virtual Environments, vol. 1, no. 2, pp. 262-271, 1992.

[32] M. Slater, “Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments,” Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1535, pp. 3549-3557, 2009.

[33] D. L. Akin, M. Minsky, E. Thiel, and C. Kurtzman, “Space applications of automation, robotics and machine intelligence systems (ARAMIS), phase 2. volume 1: Telepresence technology base development,” Technical Report NASA-CR-3734, NAS 1.26:3734, SSL-30-83, Massachusetts Inst. of Tech, Cambridge, MA, Oct. 1983.

[34] N. Polys, A. Mohammed, J. Iyer, P. Radics, F. Abidi, L. Arsenault, and S. Rajamohan, "Immersive analytics: Crossing the gulfs with high-performance visualization." Proc. 2016 Workshop on Immersive Analytics (IA), pp. 13-18, 2016.

[35] B. G. Witmer and M. J. Singer, “Measuring presence in virtual environments: A presence questionnaire,” Presence, vol. 7, no. 3, pp. 225-240, 1998.

[36] S. E. Kober, J. Kurzmann, and C. Neuper, “Cortical correlate of spatial presence in 2D and 3D interactive virtual reality: an EEG study,” International Journal of Psychophysiology, vol. 83, no. 3, pp. 365-374, 2012.

[37] I. Alsina-Jurnet and J. Gutiérrez-Maldonado, “Influence of personality and individual abilities on the sense of presence experienced in anxiety triggering virtual environments,” International journal of human-computer studies, vol. 68, no. 10, pp. 788-801, 2010.

[38] I. A. Jurnet, C. C. Beciu, and J. G. Maldonado, "Individual differences in the sense of presence." Proc. of Presence 2005: The 8th International Workshop on Presence, pp. 133-142, 2005.

[39] R. Skarbez, F. P. Brooks Jr, and M. C. Whitton, “A survey of presence and related concepts,” ACM Computing Surveys (CSUR), vol. 50, no. 6, pp. 96:1-39, 2018.

[40] M. Slater, C. Guger, G. Edlinger, R. Leeb, G. Pfurtscheller, A. Antley, M. Garau, A. Brogni, and D. Friedman, “Analysis of physiological responses to a social situation in an immersive virtual environment,” Presence: Teleoperators and Virtual Environments, vol. 15, no. 5, pp. 553-569, 2006.

[41] L. Nedel, V. C. de Souza, A. Menin, L. Sebben, J. Oliveira, F. Faria, and A. Maciel, “Using immersive virtual reality to reduce work accidents in developing countries,” IEEE computer graphics and applications, vol. 36, no. 2, pp. 36-46, 2016.

[42] S. Peltzman, “The effects of automobile safety regulation,” Journal of political Economy, vol. 83, no. 4, pp. 677-725, 1975.

[43] Y. Feng, P. Wu, G. Ye, and D. Zhao, “Risk-compensation behaviors on construction sites: Demographic and psychological determinants,” Journal of Management in Engineering, vol. 33, no. 4, pp. 04017008, 2017.

[44] X. Dong, X. Wang, R. Katz, G. West, and J. Bunting, “Fall injuries and prevention in the construction industry,” Center for Construction Research and Training quarterly data report. Silver Spring, MD, 2017.

[45] D. A. Bowman and R. P. McMahan, “Virtual reality: how much immersion is enough?,” Computer, vol. 40, no. 7, pp. 36-43, 2007.

[46] D. Brutzman and L. Daly, X3D: extensible 3D graphics for Web authors: Elsevier, 2010.

[47] w.-b. D. graphics. www.web3d.org.

ieee transactions on visualization and computer … · specifically, we used passive haptics to...

Documents