Geoscience Geoscience AustraliaAustralia

Software Software Development at the Development at the Australian Tsunami Australian Tsunami

Warning SystemWarning System

BackgroundBackground

2004 Boxing Day Tsunami.2004 Boxing Day Tsunami. Over 230,000 people die in Indian Ocean Over 230,000 people die in Indian Ocean

coastal areas.coastal areas. In 2005, Australian Government decides In 2005, Australian Government decides

to fund a national Tsunami alert system.to fund a national Tsunami alert system. The Joint Australian Tsunami Warning The Joint Australian Tsunami Warning

Centre (JATWC) was born.Centre (JATWC) was born. Became operational and launched in Became operational and launched in

2008.2008.

OverviewOverview The JATWC is a joint mission between The JATWC is a joint mission between

Geoscience Australia (GA)and the Australian Geoscience Australia (GA)and the Australian Bureau of Meteorology (BOM).Bureau of Meteorology (BOM).

GA’s mission is to detect Tsunami GA’s mission is to detect Tsunami earthquakes with the potential to impact earthquakes with the potential to impact Australia, and notify BOM and other Australia, and notify BOM and other government agencies.government agencies.

BOM’s mission is to perform Tsunami BOM’s mission is to perform Tsunami modelling, assess impact on Australia modelling, assess impact on Australia (where, how bad), and issue public and (where, how bad), and issue public and media alerts.media alerts.

Operates 24/7 year round.Operates 24/7 year round.

GA Component - ATWSGA Component - ATWS

The Australian Tsunami Warning System The Australian Tsunami Warning System (ATWS) is the GA component of the JATWC.(ATWS) is the GA component of the JATWC.

Was decided early on to use Antelope as the Was decided early on to use Antelope as the basic data flow and processing framework.basic data flow and processing framework.

Consists of an extensive software stack built Consists of an extensive software stack built on top of Antelope providing all the on top of Antelope providing all the capabilities needed to review and capabilities needed to review and communicate EQ alerts.communicate EQ alerts.

Is operated by duty seismologists who work in Is operated by duty seismologists who work in 12 hour shifts, with one person on duty at a 12 hour shifts, with one person on duty at a time.time.

ATWS A Closer LookATWS A Closer Look System is spread across several servers:System is spread across several servers:

DC – seismic data collection (orb2orb, seedlink etc.)DC – seismic data collection (orb2orb, seedlink etc.) ALERT – event processing (orbassoc, evproc etc.)ALERT – event processing (orbassoc, evproc etc.) IS – post-event review and analysis (dbloc2)IS – post-event review and analysis (dbloc2) PUB – data dissemination (orb, seedlink, autodrm)PUB – data dissemination (orb, seedlink, autodrm)

Most of our software runs on ALERT, including Most of our software runs on ALERT, including our key Decision Support Tool (DSS) .our key Decision Support Tool (DSS) .

The majority of ATWS software is written in The majority of ATWS software is written in Perl, with some components written in C.Perl, with some components written in C.

Run on Solaris/SPARC servers.Run on Solaris/SPARC servers.

ATWS DSS “orbevents”ATWS DSS “orbevents”

Main tool used to review automatic events and Main tool used to review automatic events and send out alerts.send out alerts.

Uses a manual Mwp process to provide Uses a manual Mwp process to provide magnitude estimates for EQs with mag > 6.magnitude estimates for EQs with mag > 6.

Is written in C using the Gtk+ graphics toolkit.Is written in C using the Gtk+ graphics toolkit. Sends both alerts (red) and notifications Sends both alerts (red) and notifications

(green).(green). Used in conjunction with dbloc2, with the Used in conjunction with dbloc2, with the

latter used for more in-depth review, latter used for more in-depth review, especially for small local events.especially for small local events.

ATWS DSS “orbevents”ATWS DSS “orbevents”

Manual Mwp ToolManual Mwp Tool

Other SoftwareOther Software

Heads up displaysHeads up displays Web pages that show station latency and Web pages that show station latency and

epicentres on a map.epicentres on a map. Integration with Earthquake Integration with Earthquake

CatalogueCatalogue Automatically load automatic and manual Automatically load automatic and manual

solutions into corporate Oracle database.solutions into corporate Oracle database. Automated alarm systemAutomated alarm system

Watches automatic events and send out Watches automatic events and send out emails, rings phones when triggered.emails, rings phones when triggered.

Heads up display (EQs)Heads up display (EQs)

Heads up display Heads up display (latency)(latency)

Process of ImprovementProcess of Improvement

Kent and I identified several areas of Kent and I identified several areas of improvement that were critical in supporting a improvement that were critical in supporting a mission-critical system.mission-critical system.

These includeThese include Effective use of version control and SW packaging.Effective use of version control and SW packaging. Deployment practices and configuration Deployment practices and configuration

management.management. Comprehensive system monitoring.Comprehensive system monitoring. Planned release process (Agile methodology).Planned release process (Agile methodology).

Have largely been implemented, resulting in Have largely been implemented, resulting in much better management of the system.much better management of the system.

Version ControlVersion Control

Some basic questions:Some basic questions: What is currently running in production?What is currently running in production? What was the last change made?What was the last change made?

Important to be able to answer these if Important to be able to answer these if serious about running an operational system.serious about running an operational system.

Version control is an enabler that allows them Version control is an enabler that allows them to be answered.to be answered.

Previously code was released, then sometimes Previously code was released, then sometimes checked into VC as an afterthought.checked into VC as an afterthought.

Version Control (cont)Version Control (cont)

Now all code must be checked into VC Now all code must be checked into VC before it can be released into production.before it can be released into production.

Also can see what changed by comparing Also can see what changed by comparing two previous release tags (useful for two previous release tags (useful for identifying cause of issues).identifying cause of issues).

Supports release management – lets Supports release management – lets review what changed review what changed beforebefore we release to we release to production.production.

Source code only – maintain knowledge of Source code only – maintain knowledge of how to build, in case one day we need to!how to build, in case one day we need to!

Version Control (cont)Version Control (cont)

Deployment and Deployment and ConfigurationConfiguration

Consolidated configuration module common to Consolidated configuration module common to all servers release as a Solaris package.all servers release as a Solaris package.

We use a configuration management tool We use a configuration management tool (Cfengine3) that “knows” how to copy the (Cfengine3) that “knows” how to copy the right files based on the current server to the right files based on the current server to the right spot.right spot.

Only files that have been changed are Only files that have been changed are updated.updated.

Cfengine can also restart processes if needed.Cfengine can also restart processes if needed. Also provides documentation of system Also provides documentation of system

configuration. Re-deploying a server is easy.configuration. Re-deploying a server is easy.

System MonitoringSystem Monitoring How do you know How do you know everythingeverything is running OK is running OK right right

nownow?? Comprehensive System Monitoring is key.Comprehensive System Monitoring is key. We use Nagios3 to do deep application level We use Nagios3 to do deep application level

monitoring across all processes on all servers.monitoring across all processes on all servers. Why Nagios? Standard, proven, open-source.Why Nagios? Standard, proven, open-source. We monitor the status of all application We monitor the status of all application

processes, including processes under rtexec.processes, including processes under rtexec. How? Generate the Nagios configuration from How? Generate the Nagios configuration from

rtexec files for each real-time system rtexec files for each real-time system automatically.automatically.

System Monitoring System Monitoring (cont)(cont)

System Monitoring System Monitoring (cont)(cont)

System Monitoring System Monitoring (cont)(cont)

System Monitoring System Monitoring (cont)(cont)

Development processDevelopment process

Tools and technologies don’t do much alone.Tools and technologies don’t do much alone. They exist to support a process that ensures They exist to support a process that ensures

they are effectively put to good use.they are effectively put to good use. A development methodology is this process.A development methodology is this process. We follow an Agile methodology, loosely We follow an Agile methodology, loosely

based on Scrum.based on Scrum. Regular, planned release cycle or “sprints”.Regular, planned release cycle or “sprints”. Items of work are assigned to sprints.Items of work are assigned to sprints. Unassigned items are used to plan next Unassigned items are used to plan next

sprint.sprint.

The Future The Future

Redevelopment of all ATWS software, Redevelopment of all ATWS software, including new DSS which will be a major including new DSS which will be a major undertaking.undertaking.

Possibly move to Python as our main Possibly move to Python as our main development language.development language.

Migrate to Apple Mac or Linux in next few Migrate to Apple Mac or Linux in next few years.years.

Array processing.Array processing. New magnitude calculators.New magnitude calculators. Regional responsibilities.Regional responsibilities.

DSS RedevelopmentDSS Redevelopment

New heads up displays.New heads up displays. More information – answer the most More information – answer the most

common questions of visitors:common questions of visitors: When was the last Earthquake?When was the last Earthquake? Who is on duty?Who is on duty? When was the last alarm/alert sent?When was the last alarm/alert sent? What did the PTWC get for this event? What did the PTWC get for this event?

Less distractions for duty officer.Less distractions for duty officer. Extra information sources, such as Extra information sources, such as

Twitter.Twitter.

DSS Redevelopment DSS Redevelopment (cont)(cont)

DSS Redevelopment DSS Redevelopment (cont)(cont)

New “orbevents”.New “orbevents”. Much greater functionality.Much greater functionality. Seismic awareness, i.e. show detections on Seismic awareness, i.e. show detections on

map.map. Less clutter, focused on primary mission.Less clutter, focused on primary mission. Timeline – clear indicator of how long left to Timeline – clear indicator of how long left to

make decision, and when different events make decision, and when different events occurred (origin, magnitude, etc).occurred (origin, magnitude, etc).

External data – Twitter, felt reports, external External data – Twitter, felt reports, external solutions. Increase situational awareness.solutions. Increase situational awareness.

DSS Redevelopment DSS Redevelopment (cont)(cont)

DSS Redevelopment DSS Redevelopment (cont)(cont)

Python Python

Perl has been main language of choice.Perl has been main language of choice. My background is in Perl.My background is in Perl. So why even think about Python?So why even think about Python? Actually lots of reasons, but 2 main ones:Actually lots of reasons, but 2 main ones:

Finding good Perl developers is hard. Really Finding good Perl developers is hard. Really hard. No-one wants to learn Perl – existing hard. No-one wants to learn Perl – existing Perl developers want to learn something else!Perl developers want to learn something else!

Lack of bindings to modern Graphics Toolkits.Lack of bindings to modern Graphics Toolkits.

Python (cont)Python (cont) We expect the new DSS to rely heavily on We expect the new DSS to rely heavily on

advanced graphics manipulation that need advanced graphics manipulation that need canvas like functionality.canvas like functionality.

Gtk+, does not have a native canvas element. Gtk+, does not have a native canvas element. An extension, Goocanvas, is available, but An extension, Goocanvas, is available, but

Perl bindings are incomplete compared to Perl bindings are incomplete compared to Python.Python.

Also, most modern cross-platform Toolkit Also, most modern cross-platform Toolkit available, Qt4, has no Perl bindings but available, Qt4, has no Perl bindings but excellent Python bindings.excellent Python bindings.

Tk? Possibility, but seems like a step Tk? Possibility, but seems like a step backwards.backwards.

Python (cont)Python (cont) Other benefits:Other benefits:

Simply, easy to read and maintain. Simply, easy to read and maintain. Can focus on the problem, not the language.Can focus on the problem, not the language. Becoming the worlds Becoming the worlds de factode facto standard scripting standard scripting

language, bindings to just about everything.language, bindings to just about everything. Great, active worldwide community. Particularly Great, active worldwide community. Particularly

in scientific communities, including seismic ones in scientific communities, including seismic ones (obspy).(obspy).

Large developer community within GA.Large developer community within GA. Amazing range of first-rate libraries: numpy, scipy, Amazing range of first-rate libraries: numpy, scipy,

matplotlib, obspy, PyQt4, PIL, and many more…matplotlib, obspy, PyQt4, PIL, and many more… No (or little) dependency hell.No (or little) dependency hell.

Python (cont)Python (cont) Issues?Issues?

Antelope bindings, unclear future.Antelope bindings, unclear future. Multi-threading – works but issues (the (in)famous Multi-threading – works but issues (the (in)famous

GIL).GIL). Staff retraining, recruitmentStaff retraining, recruitment

Overall we think that the benefits speak for Overall we think that the benefits speak for themselves.themselves.

Have already started prototyping application Have already started prototyping application for the new DSS in Python using Kent’s for the new DSS in Python using Kent’s contributed Python bindings for Antelope.contributed Python bindings for Antelope.

Settling on the Qt graphics framework in Settling on the Qt graphics framework in combination with Python.combination with Python.

Python (cont)Python (cont)

Mac vs. LinuxMac vs. Linux Big question running through the Antelope Big question running through the Antelope

community.community. Antelope on Solaris has been EOL’d (2013), Antelope on Solaris has been EOL’d (2013),

future Linux support uncertain.future Linux support uncertain. We have a corporate preference for Linux, as it We have a corporate preference for Linux, as it

plays well in an Enterprise environment.plays well in an Enterprise environment. Lack of Mac expertise, and questions remain Lack of Mac expertise, and questions remain

about how Mac’s can operate in a data-centre about how Mac’s can operate in a data-centre context.context.

Redundant power supplies, LDAP, SAN, Lights-Redundant power supplies, LDAP, SAN, Lights-out management (LOM), virtualization, updates.out management (LOM), virtualization, updates.

Magnitude CalculatorsMagnitude Calculators We currently have customized Ml, Mb, Ms (Msx) We currently have customized Ml, Mb, Ms (Msx)

modules running under orbevproc.modules running under orbevproc. We also use the standard AutoMwp calculator.We also use the standard AutoMwp calculator. CMT process works outside of orbevproc (trexerpt).CMT process works outside of orbevproc (trexerpt). Want to develop MT, Theta, MCoda, WPhase, etc.Want to develop MT, Theta, MCoda, WPhase, etc. New orbevproc model for running these modules:New orbevproc model for running these modules:

Let orbevproc gather, filter and process waveforms.Let orbevproc gather, filter and process waveforms. Write to SAC files, manipulate SAC headers.Write to SAC files, manipulate SAC headers. Call external “black-box” module to run against SAC files.Call external “black-box” module to run against SAC files.

Questions?Questions?