2012 SCEC Report

Enabling the SCEC Modeling Community to Conduct Physics-Based, Deterministic Broadband Earthquake Simulations using a High-Performance Portable Parallel Software

PI: Jacobo Bielak

Co-PI: Ricardo Taborda

Computational Seismology Laboratory

Department of Civil and Environmental Engineering

Carnegie Mellon University

1

Enabling the SCEC Modeling Community to Conduct Physics-Based, Deterministic Broadband Earthquake Simulations using a High-Performance Portable Parallel Software

1. Project Summary

The main goal of this project was to prepare a tool-framework that would enable SCEC’s modeling com-munity and outside partners to conduct physics-based, deterministic earthquake simulations using Hercu-les—the octree-based finite-element portable parallel software for seismic wave-propagation problems developed by the Quake Group at Carnegie Mellon University (CMU) (Tu et al., 2006; Taborda et al., 2010). To that end, during the last year we have taken a series of steps to ensure future use of Hercules beyond its group of developers at CMU. These activities include: (1) moving the Quake Group Web site and computer codes archive from a private to a public space; (2) porting Hercules source code from a pri-vate CVS-based version control system, with a CMU-access only repository, to a Web-based hosting service using the Git revision control system, with a membership access repository; (3) providing an issue tracker and a Wiki documentation infrastructure for supporting the code; and (4) creating a light version of Hercules (Hercules-LT) for non-developers. We expect some of these steps to continue to be a work in progress beyond the past year’s support for this project, and we are pleased to report to SCEC that all the related activities were completed or well-put in motion by the end of this period. More important, these steps constitute the completion of a first stage in preparation of Hercules as a community code that will enable the SCEC modeling community to conduct physics-based, deterministic simulations at increas-ingly higher frequencies (~5 Hz) in the near future. Other impacts of bringing Hercules up to a more open framework for community development is its potential future use in interdisciplinary projects such as CyberShake and the Broadband Simulation Platform, or its use as a reference tool for other groups working on simulation, tomography, and verification and validation of the various source and velocity models available through SCEC.

2. Activities Report

2.1. A New Public Space

Moving Hercules code repository to a new site also implied finding a new space for the Quake Group’s computational resources and documents archive. In the past, the group’s Web site and code repository had been hosted at CMU’s School of Computer Science (CS). Though the Web site was publicly acces-sible, the repository and archive were not. To facilitate the group’s operation, we moved most of the ma-terial from the CS location to a new hosting space at CMU’s Civil and Environmental Engineering De-partment (CEE) and will continue to transfer data over the following months. As part of this effort, we redesigned the group’s Web site, now accessible at http://quake.ce.cmu.edu/. Figure 1 shows both the old and new sites. The new hosting site is also being used to facilitate data sharing with other members of the SCEC community. We have set up an “exchange” space within the new site that is being used for that purpose. During this year, for instance, we shared simulation synthetics and data seismograms from our 4-Hz Chino Hills simulation (Taborda and Bielak, 2013) with two groups within SCEC using this portal (see Fig. 2).

2

Figure 1. Screenshots of the old and new Web site portals of the Quake Group.

Figure 2. Screenshot of the exchange site created within the new group’s space to facilitate data sharing.

2.2. Porting Hercules to a New Repository

Since 2005, Hercules’ source code had been hosted in a private repository supported by CMU’s CS com-puting services. That repository used the Concurrent Versioning System (CVS) as the client-server soft-ware for keeping track of the code’s changes and contributions by the group’s developers. Although CVS had been very convenient for tracking the code’s evolution over the last eight years, CS hosting offered access to members of the CMU group only, and setting up access for other contributors outside CMU was onerous and complicated. Therefore, to facilitate contributions from future SCEC and outside developers and/or authorized users, and as part of moving the group’s resources to a new location, we ported our CVS repository outside CS to a temporary local host in CEE. Then, after the initial phase of changing to the new Web site was completed, we ported the code to the Git version control system (http://git-scm.com/) and set up a new repository in GitHub (https://github.com). GitHub is a Web-based hosting service for software development of projects that use Git as their version control system. The combina-tion of using Git and GitHub offers larger flexibility for hosting Hercules and makes significantly easier the future participation of other users outside the CMU Quake Group. This step was crucial for achieving the main goal of the project. Another great advantage that comes with the decision of porting the code to this system is that Git is a distributed version control system and not a centralized one. This means that the code history is kept by all developers at all times, and can be recovered from any copy of the reposi-tory. Git is also more flexible for branching the code and keeping track of multiple variations, and the GitHub portal allows tracking issues and providing documentation through a Wiki (see next section). These capabilities ensure better redundancy of the code-tracking evolution and provide a sustainable software infrastructure. Figure 3 shows screenshots of the group’s GitHub space, its Hercules repository and the current members—which we hope to grow as the code is advertised within SCEC. Figure 4 shows a screenshot of the repository’s files.

3

Figure 3. Screenshots of the CMU-Quake hosting space in GitHub’s Web server.

Figure 4. Screenshot of the Hercules repository in GitHub’s CMU-Quake code hosting space.

2.3. Creating a Hercules Wiki on GitHub

As mentioned above, one of the advantages of porting Hercules to GitHub was the possibility of creating a Wiki for documenting the code and offering users a system for tracking issues. We have created Wiki pages that describe Hercules and how to download, configure, and compile the code in different systems. Figure 5 shows a screenshot of the Wiki page with the general description of the code. Although there is still significant amount of information to be uploaded before being able to provide full support to outside users, having this space will facilitate a sustainable framework for the code’s evolution. We will continue to expand the documentation about the code progressively. In addition, next to the Wiki, GitHub offers a system for tracking issues. Figure 6 shows a screenshot of this feature. We are not yet tracking any code issue but we believe this feature will prove very useful to other SCEC users in the future.

4

Figure 5. Screenshot of the Hercules Wiki in GitHub’s CMU-Quake code hosting space.

Figure 6. Screenshot of the issue tracking system in GitHub’s for Hercules.

2.4. Hercules LT

The last main step of the project was to prepare a light version of the code for most non-specialized users. Members of the Quake Group at CMU constantly work on new developments to improve and expand the code’s capabilities, efficiency and accuracy. However, as new features are being developed, they tend to be unstable or have the potential of altering the standard behavior of the code and accuracy of its results. To facilitate the use of the code by other users outside the group of specialized developers, we have cre-ated a branched version of Hercules that will operate as the default light-version of the code. Figure 7 shows the different branches of the code with all the current features in development by the members of the Quake Group and the light version of the code that will operate as the default mode for most users.

3. References

Taborda, R., and Bielak, J., (2013). Ground-motion simulation and validation of the 2008 Chino Hills, California, earthquake. Bull. Seismol. Soc. Am., 103(1):131–156.

Taborda, R., Lopez, J., Karaoglu, H., Urbanic, J., and Bielak, J. (2010). Speeding up finite element wave propagation for large-scale earthquake simulations. Technical Report CMU-PDL-10-109, Carnegie Mellon University, Parallel Data Lab.

5

Tu, T., Yu, H., Ramirez-Guzman, L., Bielak, J., Ghattas, O., Ma, K.-L., and O’Hallaron, D. R. (2006). From mesh generation to scientific visualization: An end-to-end approach to parallel supercomputing. In SC’06: Proceedings of the 2006 ACM/IEEE International Conference for High Performance Com-puting, Networking, Storage and Analysis, page 15, Tampa, Florida. IEEE Computer Society.

Figure 7. Current branching of Hercules in development (for developers) and Hercules LT version (for general users).