OpenMP for Networks of SMPs
Y. Charlie Hu, Honghui Lu, Alan L. Cox, Willy Zwaenepoel
ECE1747 – Parallel Programming
Vicky Tsang
Background
Published in the Journal of Parallel and Distributed Computing, vol. 60 (12), pp. 1512-1530, December 2000
Work to further improve TreadMarks Presents an alternative solution to MPI
Roadmap
Motivation Solution OpenMP API TreadMarks OpenMP Translator Performance Measurement Results Conclusion
Motivation
To enable the programmer to reply on a single, standard, shared-memory API for parallelization within and between multiprocessors.
To provide another standard other than MPI?
Solution
Presents the first system that implements OpenMP on a network of shared-memory multiprocessors
Implemented via a translator converting OpenMP directives to calls in modified TreadMarks
Modified TreadMarks uses POSIX threads for parallelism within an SMP node
Solution
Original version of TreadMarks:A Unix process was executed on each
processor of the multiprocessor node and communication between processes was achieved through message passing
Fails to take advantage of hardware shared memory
Solution
Modified version of TreadMarks POSIX threads used to implement parallelism OpenMP threads within a multiprocessor share a
single address space Positive:
Reduces the number of changes to TreadMarks to support multithreading on a multiprocessor
OS maintains the coherence of page mappings automatically Negative:
More difficult to provide uniform sharing of memory between threads on the same node and threads on different nodes
OpenMP API
Three kinds of directives: Parallelism/work sharing Data environment Synchronization
Based on a fork-join model Sequential code sections executed by master
thread Parallel code sections are executed by all
threads, including the master thread
OpenMP API
Parallel directive – all threads perform the same computation
Work sharing directive – computation is divided among the threads
Data environment directive – control the sharing of program variables
Synchronization directive – control the synchronization between threads
TreadMarks
User-level SDSM system Provides a global shared address space
on top of physically distributed memories Key functions performed are memory
coherence and synchronization
TreadMarks – Memory Coherence
Minimize the amount of communication performed to maintain memory consistency by: a lazy implementation of release consistency reducing the impact of false sharing by allowing
multiple concurrent writers to modify a page
Propagation of consistency information is postponed until the time of an acquire
TreadMarks - Synchronization
Barrier implemented as acquire and release messages
Governed by a centralized manager
TreadMarks – Modifications for OpenMP Inclusion of two primitives:
Tmk_fork Tmk_join
All threads created at the start of a program’s execution to minimize overhead.
Slave threads are blocked during sequential execution until the next Tmk_fork is issued by the master thread.
TreadMarks – Modifications for Networks of Multiprocessors POSIX thread enabled sharing of data between processors.
Addition of some data structures, such as message buffers, in thread-private memory for data that is to remain private within a thread.
A per-page mutex was added to allow greater concurrency in the page fault handler.
Synchronization functions in TreadMarks were modified to use POSIX thread-based synchronization between processors within a node and existing TreadMarks synchronization functions between nodes.
A second mapping was added for the memory that is shared between nodes so shared-memory pages can be updated while the first mapping remains invalid until the update is complete. This reduces the number of page protection operations performed by TreadMarks.
OpenMP Translator
Synchronization directives translate directly to TreadMarks synchronization operations.
The complier translates the code sections marks with parallel directives to fork-join code.
Data environment directives implemented to work with both TreadMarks and POSIX threads, hiding the interface issues from the programmer.
Performance Measurement
Platform IBM SP2 consisting of four SMP nodesPer node:
Four IBM PowerPC 604 processors 1 GB memory Running AIX 4.2
Performance Measurement
ApplicationsSPLASH-2 Barnes-HutNAS 3D-FFTSPLASH-2 CLUSPLASH-2 WaterRed-Black SORTSPModified Gramm-Schmidt (MGS)
Results
Results
Results
Results
Conclusion
Enables the programmer to rely on a single, standard, shared-memory API for parallelization within and between multiprocessors.
Using shared hardware memory reduced data and messages transmitted.
The speedups of multithreaded TreadMarks codes on four four-way SMP SP2 nodes are within 7-30% of the MPI versions.
Critique
Solution allows easier implementation of program parallelization across multiprocessors if speedup is not crucial
OpenMP is easier on the programmer but speedup still not as good as MPI
Critique
Issues: AIX has inefficient implementation of page protection
Paper claims that every other brand of Unix, including Linux, uses data structures that handle mprotect operations more efficiently
Why wasn’t the solution implemented on another platform?
Paper failed to present a big motivation for using this solution over MPI.
Thank You