icccbe2010© Nottingham University PressProceedings of the International Conference on Computing in Civil and Building Engineering W Tizani (Editor)

Abstract This paper presents a new technology and a systematic approach for disaster response and recovery of Critical Physical Infrastructures (CPIs). Our suggested approach is based on using a Mobile Workstation Chariot (MWC) assembled on Segway personal transporter which supports both horizontal and vertical real-time visual data capture and transmission flow, first responders and civil engineers can quickly traverse hazardous terrain, collect and transmit photographs/videos, and communicate with the command center in real-time. Using MWC wireless communication tools, first responders and civil engineers can access disaster-survivable black boxes allowing Building Information Models (BIM), pre-disaster photographs and operational information of buildings to be collected and communicated back to the command center. Finally at the command center, using sensed visual data and image-based reconstruction techniques, the post-disaster site is reconstructed in 3D. The resulting integrated representation of the post-disaster model and the collected photographs are superimposed over the pre-disaster BIM to generate a 4D Augmented Reality (D4AR) model. By integrated representation of pre-disaster and post-disaster information, the D4AR allows damages, safety and stability of the CPIs as well as possible rescue operation routings and plans to be assessed. Critical information for disaster response and recovery can be analyzed and communicated back to the field easily and quickly. We present preliminary results of our experiments for collecting, analyzing, and visualizing sensed data using the MWC as well as the D4AR. These results demonstrate a great potential for application of MWC and D4AR for disaster response and recovery operations. The limitation and benefits of this approach plus further required developments are discussed.

Keywords: disaster preparedness, response and recovery, augmented reality, 4D, mobile workstation

1 Introduction Concerns about eXtreme Events (XEs) such as natural disaster (e.g., hurricanes, cyclones, earthquakes) and accidental or intentional man-made disasters (e.g., fires and terrorist attacks) are becoming increasingly relevant as populations of urban areas are increasing. In this new climate, it is likely we will encounter XEs more frequently and their impact on our built environment will be more severe. The future of our cities will be influenced by the manner in which these events are addressed. Given the growing complexity and increasing inter-dependence of modern urban infrastructure elements, a paradigm shift is required in the make-up of first responder teams to include civil engineers (Peña-Mora et al. 2008). First responders and civil engineers need to be equipped with data

Remote assessment of pre- and post-disaster critical physical infrastructures using mobile workstation chariot and D4AR models

M Golparvar-Fard & J Thomas University of Illinois, Urbana-Champaign, IL, USA

F Peña-Mora Columbia University, New York, NY, USA

S Savarese University of Michigan, Ann Arbor, MI, USA

collection, processing, and communication technologies for more effective deployment of response, damage assessment, and rescue and recovery planning of the affected areas.

Figure 1. An overview of the proposed approach and data flow. Data collection, analysis and communication components are also shown.

In this paper, we present a new technology and a systematic approach for building stability assessment and rescue operation routings in disaster response and recovery of Critical Physical Infrastructures (CPIs). Our suggested approach is structured on a personal transporter utilizing a commercially available Segway which has been designed to carry a payload of information collection and communication equipment (tablet PC, still and video/image capture devices, Global Positioning System (GPS) receivers, and other additional communication devices). This Mobile Workstation Chariot (MWC) enables the first responder to effectively traverse hazardous terrain and communicate in real-time with the command center to facilitate optimal disaster response and recovery. Using these wireless communication tools, first responders and civil engineers can access disaster-survivable black boxes pre-located in buildings, allowing Building Information Models (BIM), pre-disaster photographs and operational information of buildings to be collected and communicated back to the command center. Images captured in the field from the MWC will provide technicians at the command center with information to automatically generate a virtual 3D reconstruction of the post-disaster site using image-based reconstruction techniques. The resulting integrated representation of the post-disaster reconstruction and the collected photographs are superimposed over the pre-disaster building information model (BIM) to generate a 4D Augmented Reality (D4AR) model. Through integrated representation of pre-disaster and post-disaster information, the D4AR allows the stability and safety of the CPIs as well as possible rescue operation routings and plans to be interactively assessed. Critical information for disaster response and recovery can be analyzed and communicated to the field easily and quickly. Fig.1 shows an overview of the proposed approach, data flow, as well as its technological components. The specific contributions of this work and our new hardware/software components, i.e. MWC and D4AR - which are specifically designed and developed to facilitate visual data collection, stability analysis, and communication - are presented in detail here. An overview of interactions of these components within the proposed system is discussed in a case study of preliminary field trials.

2 Mobile Workstation Chariot A self-balancing, two-wheeled Segway personal transporter has been modified by adding a structural framework which provides mounting support for a tablet PC, still and video cameras, Global

Positioning System receivers and other communication devices, creating a Mobile Workstation Chariot (MWC). Real time connectivity support is provided to facilitate data communication to and from first responders in the field. The design provides multiple positions where the PC and other additional communication devices may be mounted simultaneously. The structural Chariot platform is created from an aluminum framework which has three levels for supporting equipment. Movable mounting connectors allow commercially available tripod heads mounted on telescoping columns to be placed in a variety of positions to locate the communications equipment conveniently around the Segway – on either side and across the front. Fig. 2a shows several configurations of the data gathering and communication apparatus on the MWC. The columns can be raised and lowered or mounted horizontally while the ball head tripod mounts provide a full 360° rotation in the horizontal and vertical axes. These adjustments allow the equipment to be positioned normal to the view or appropriate to the user to provide the best orientation to the first responder. By reorienting the PC, a group of first responders standing next to the MWC can use it as a collaborative workstation (Fig. 2b).

Figure 2. (a) MWC provides the ability to carry and use data collection and communication equipment during rapid response to XEs. (b) First responder can quickly traverse hazardous terrain, and also use the MWC as a collaborative workstation in the field.

Counterbalancing weights may be added to the lower rod of the framework, generally near the back of the unit, to help equalize the weight distribution of the communications equipment. While providing maximum flexibility for mounting of these important tools, the Chariot framework surrounding the user provides a psychological sense of safety. The structure is designed to accommodate the range of motion of the user while not impairing movement of the Segway PT steering handle. Using the MWC, the first responder quickly moves deep into the disaster field, collecting visual data from many viewpoints as he/she drives MWC around and through the site. As the MWC moves within close proximity of buildings wherein building black boxes are pre-located, data (BIM) is wirelessly collected on the tablet PC for transmission to the Command Center. GPS receivers can location stamp critical information that is gathered. Although the Segway provides transportation for a single person, the MWC can become a mobile workstation/communication center in the field, supporting a group of first responders and civil engineers (Fig. 2b).

3 D4AR model In this component as shown in Fig. 3, using the images and video streams that are collected with the MWC, the observed site is automatically reconstructed. The reconstructed observed model along with the BIM that is extracted from the pre-located building black box is used to generate a D4AR model for visualizing observed and expected models plus structural deviations in between. In the following, an overview of the steps that are required to generate such D4AR models is presented:

First, using the images and video streams that are collected and transmitted with the MWC and based on the underlying Structure-from-Motion (SfM) technique of the D4AR (Golparvar-Fard et al. 2009, Snavely et al. 2008) we automatically generate an underlying 3D geometry for the observed site.

(b) (a)

In this case, cameras are not pre-calibrated as we are interested to let first responders easily modify the configuration of the camera (i.e., zoom and focus settings) through the tablet PC’s interactive system. In addition, we allow other images that are captured by first responders’ handheld cameras to be used to improve the observed model reconstruction in inaccessible areas. Compared to Golparvar-Fard et al. (2009), in our algorithm the computation performance of the k-d tree feature matching scheme for sensed images is improved by only allowing searching and matching to be conducted in smaller image subsets; i.e., we only allow each image (Ii) to be automatically matched with all images within a proximity of δ to the image (i-δ, i+δ). This approach specifically improves the quality of reconstruction on reflective surfaces (e.g., curtain walls or metal facade claddings) as it does not allow images with wide baselines to be used for matching. Such images especially when they are taken in a sequence may incorrectly represent consistent matches across image pairs, ultimately leading to noisy point cloud reconstructions. Overall through this step, the underlying geometry of the scene is automatically reconstructed and cameras are calibrated.

Figure 3, An overview of data and process in the D4AR modeling, processed at the remote command center.

Next, the expected BIM model is superimposed over the integrated observed point cloud and the camera model. In cases where GPS may not be operable, we allow registration to be conducted interactively. By registration, we upgrade the observed point cloud to its Euclidean dimensions. The Euclidean model along with camera calibration parameters is fed into a Multi-view Stereo algorithm (Furukawa and Ponce 2009) to improve the observed model reconstruction. At this stage, the integrated observed and expected scene is quantized into voxels (small volumes in space). By using a voxel coloring and labeling algorithm developed in Golparvar-Fard et al. (2010), which traverses the quantized scene and checks consistency in visual appearance, the observed site is volumetrically reconstructed and labeled for observed occupancy. Using a similarly structured voxel coloring and labeling algorithm, the expected scene is also labeled for occupancy and expected visibility. These two labeled observed and expected voxels are fed into a novel Bayesian model to identify the ratio of occupied voxels (voxels that contain building elements) within the expected visible area. The results are dynamically classified with a Support Vector Machine (SVM) classifier to reason about missing voxels and their corresponding BIM elements. This step classifies voxels into (1) [Occupied | Empty | Occluded] observed and (2) [Occupied | Visible] expected voxels and further identifies two critical types of areas within the scene: (1) Expected-Empty areas (damaged areas wherein components were expected to be observed, but are not), which their identification is particularly useful for structural stability analysis and identification of structural drifts due to disaster; and (2) Not Expected-Occupied areas (detected areas wherein damaged parts of the building or blockages may hinder rescue operations), which their identification is useful for rescue operation routing. Finally, the detected observed model, camera configurations, the expected model, and labeled voxels are all fed into the D4AR viewer to visualize observed and expected models plus deviations in an integrated fashion. In this case, deviations are color-coded based on a traffic light metaphor (Golparvar-Fard et al. 2009b) highlighting areas or components which need further consideration and should be communicated between decision-makers and first responders.

4 Experiments In order to verify the developed hardware/software configuration, we preliminarily tested it in a trial on a building currently in use. In our trial, first responders traversed the site and with the still camera - which is mounted on the MWC - captured 163 images from the exterior of the building under study (at rate of ~5 image/second and 13M pixel resolutions). A BIM model for the building was also generated and made accessible for integration with the observed point cloud. We synthetically reduced the quality of the images to ~4M Pixels on the tablet PC to demonstrate that our system can work with lower image resolutions. Although reducing image resolution reduces the number of feature points that could be detected, yet this can expedite data transmission and reduce the computation cost for constructing observed scenes. By conducting reconstruction quality vs. computation cost analysis, the image proximity parameter (δ) was set to 5. Fig. 4a shows the responder and Fig. 4b to 4f highlight the building that was captured while traversing the scene. Fig. 4g visualizes the fully automatically reconstructed model along with the path on which the MWC traversed the scene to capture the observed model (4b). In this case, the camera was controlled through the interface of the tablet PC mounted on the MWC. Fig. 4h shows the location of camera frustum along the traversing path. In Fig. 4i-k, synthetic reconstruction and image-based rendered views are shown.

Figure 4. (a) First responder collecting images through the tablet PC mounted on the MWC. (b-f) Still images sequentially captured for reconstruction of the case study building exterior.(g-k) Reconstruction synthetic views.

Once the scene was reconstructed, the observed model was registered with the BIM model (Fig. 5). Since there is a strong possibility for significant changes to a site during disaster, finding similar features for automated registration of the BIM model with the underlying reconstructed scene is not always feasible. Therefore at this stage, we also allow users to interactively select a set of correspondence data points between the BIM and the underlying reconstructed scene. Subsequently the reconstructed scene is improved with Multi-view stereo and is fed into observed and expected voxel coloring and labeling steps to identify and highlight different voxel types.  

Figure 5, (a – c) Synthetic views of the D4AR model. (d) Camera frustum in vicinity of the CPI. (e-g) D4AR model viewed from the camera in (d). (h) BIM information queried and visualized.

 

Finally, the reconstructed scene and BIM are fed into the D4AR viewer, allowing as-observed and as-expected models to be visualized simultaneously. Within the integrated scene, deviations can be easily detected which further supports safety and structural stability decision-making tasks (Fig. 5). The computational results of our experiment are reported in Table 1. In this case, the computation time can be significantly improved if feature detection and matching steps are conducted in parallel. Table 1. Results of computation time, reconstruction density and registration error. # images

data collection time*

SfM Reconstr. time

D4AR Reconstr. time

SfM # points

D4AR # points

Registration error BIM + point cloud

163 5.0 min 2 hr 49 min 3 hr 56 min 119,633 398, 524 37 mm * Including transmission time for photos + BIM.

5 Summary and future work This systematic disaster response and recovery approach proposes a Mobile Workstation Chariot for first responders and civil engineers which enables them to quickly traverse hazardous terrains, collect and transmit photographs/videos, and communicate with the command center in real-time. Using MWC wireless communication tools, disaster-survivable black boxes can be accessed; BIMs can be collected and communicated back to the command center. At the command center, the post-disaster site is reconstructed using sensed visual data. The resulting integrated representation of the post-disaster and pre-disaster models are superimposed over one another and generate D4AR models. The resulting D4AR models allow safety and stability of the CPIs as well as possible rescue routings to be remotely assessed. Preliminary results of our experiments for collecting, analyzing, and visualizing sensed data using the MWC and the D4AR models demonstrate a great potential for remote building assessment and developing rescue operation routings. We are currently working on shape modeling techniques to automatically generate surfaces from post-disaster point clouds and use those towards automating stability analysis. Further experiments on disaster training sites are being conducted.

Acknowledgements This research was support by NSF grants CMS-0427089 and CMMI-0800500. The authors also want to thank S. Roh, K. Surheyao, J. Celis, L. de Pombo and J. Winston who have participated in the trial. Finally, we would like to thank R. West for his help in fabricating the Chariot and Illinois Fire Institute (IFSI) for allowing us to conduct experiments at the disaster site and their valuable support.

References FURUKAWA, Y. and PONCE, J., 2009. Accurate, dense, and robust multi-view stereopsis. IEEE PAMI. GOLPARVAR-FARD, M., SAVARESE, S., and PEÑA-MORA, F., 2010. Automated model-based recognition of progress

using daily construction photographs and IFC-based 4D models. Proc., Construction Research Congress. GOLPARVAR-FARD, M., PEÑA-MORA, F., and SAVARESE, S., 2009a. D4AR- A 4-Dimensional Augmented Reality

model for automating construction progress data collection, processing and communication. J. of ITcon, 14, 129-153. GOLPARVAR-FARD, M., PEÑA-MORA, F., ARBOLEDA C.A., and LEE S.H., 2009b. Visualization of construction

progress monitoring with 4D simulation model overlaid on time-lapsed photographs. ASCE Journal of Computing in Civil Engineering, 23 (6), 391-404.

SNAVELY, N., SEITZ, S.M., and SZELISKI, R., 2008. Modeling the world from internet photo collections. International Journal of Computer Vision, 80(2), 189–210.

TSAI, M., LIU, L., PEÑA-MORA, F., and ARBOLEDA, C.A., 2008. A preliminary design of disaster-survivable building blackbox system for urban disaster response. Journal of ITcon, 13, 179-192.

PEÑA-MORA, F., AZIZ, Z., CHEN, A.Y., PLANS, A., FOLTZ, S., 2008. Building assessment during disaster response and recovery. Urban Design and Planning, 161, 183-195.