MPI Message Queue Debugging Interface

Chris GottbrathDirector, Product Management

2TotalView Technologies –Proprietary– Plans Subject to Change without Notice

MPI Message Queue Helper Library

• Provides access to MPI message queue state at a point in time– Within one paused, deadlocked, hung, or crashed MPI process

• Certain TotalView functions apply this API across many processes to gather a global view of MPI process queue state within an MPI job

– For troubleshooting, debugging and diagnostic purposes• Message Queue Model

– Three “conceptual queues” • MPI Send Queue• MPI Receive Queue• MPI Unexpected Queue

– MPI can provide info for any of all of these queues– The MPI may have more or less (or different) queues– Message traffic that is part of collective operations may or may not be

included• Data Presented includes

– local communicator handle, local and remote rank id, message tag, message data or destination buffer into which the data will be written

3TotalView Technologies –Proprietary– Plans Subject to Change without Notice

MPI Message Queue Helper Library

• Helper library– Provided by MPI vendor– Loaded by debugging tool

• Tool provides basic services via callback functions– Locating symbols in target – Reading data from target

• Library encapsulates MPI library specific details– Isolates debugging tools from MPI library internals– Multiple library version compatibility with the same debugging tool– New MPI libraries can be supported with no tool changes

• Advantages of library implementation– Live target or post-mortem corefile target– Doesn’t require target to make progress – Target types are explicitly declared so the target architecture type can

differ from that of the library and of the debugger (which do need to match)

4TotalView Technologies –Proprietary– Plans Subject to Change without Notice

Message Queue Helper Library

• Current Status– Wide industry acceptance

• Numerous MPI implementations have adopted this strategy– MPICH 1 and MPICH 2– Open MPI– Intel MPI– IBM POE– SGI MPT– MVAPICH– Sun Cluster Tools

• Library is used by multiple debugging tools• Open source & proprietary closed source

– Publicly documented• Cownie and Gropp, EuroPVM/MPI 1999• Multiple open source implementations serve as further documentation

• Standardization– Should this be part of the MPI standard?