Next: Example Up: Implementation Previous: Mesh Partitioning

Communication

The proposed approach has been successfully implemented into an existing object oriented finite element environment [8]. The program structure has been designed to be independent of a particular message passing library. Currently, the MPI message passing library [7] has been used. It is supported by many existing platforms including workstation clusters. All communication is buffered by the finite element code, not by the message passing library. Explicit buffering by the application leads to an effective memory usage and efficient data exchange, but requires synchronization between collaborating processes. A nonblocking point to point communication has been used. It allows the data exchange to be performed by a specialized hardware (if installed), providing the possibility for some computation to be done before the communication completion.

In order to hide implementation details concerning the particular message passing library, a representation for Communication buffer has been introduced. It is an abstraction for application communication buffer. It provides services for buffer initialization and resizing as well as methods for packing/unpacking data to/from the buffer. Multiple messages can be packed/unpacked into/from a single buffer. The services for packing/unpacking take care about multiple messages stored in the buffer, they maintain proper current buffer position for data insertion/retrieval. Multiple services, allowing to send/receive a buffer to/from a selected destination, are also declared, providing functionality common across many message passing libraries. The implementation of these services is left on derived classes, which will represent interfaces to the particular message passing libraries.

All communication with a particular remote partition is controlled by Domain communicator. This class maintains its communication rank, communication maps, and related Communication buffers.

Communication maps, which can be thought as lists of entities (nodes, elements) that participate in the communication, are established according to the mesh partitioning prior the actual analysis separately for send and receive operations. While the send and receive maps are identical for the node-cut strategy, they are different in the case of the element-cut strategy. It should be also recalled that communication maps for node-cut approach (exchange of shared node data) can be assembled locally on each partition directly from the mesh partitioning data. However, this is not true for the element-cut approach, where only the receive map can be setup locally. The send map must be established by communication in terms of mutual exchange of receive maps between collaborating processes and consequent selection of requests related to each local partition. The corresponding local send and remote receive (and vice versa) maps must be uniquely ordered to guarantee correctness of packing and unpacking operations. In general, the communication maps can vary dynamically during the analysis to reflect the potential repartitioning, for example due to the recovery of the load balance.

In order to fully exploit the features of the nonblocking communication scheme, the Communication buffers are provided separately for send and receive operations. It should be emphasized that there are separate instances of Domain communicator on each partition, each one dedicated for communication with a particular remote partition. This essentially enables an overlapping data exchange between partitions (making the message passing very efficient).

The Domain communicator is responsible for packing and unpacking relevant data. A general implementation of these operations has been obtained using call-back procedures, which pack and unpack the data according to the communication maps using the elementary services provided by the Communication buffer.



Next: Example Up: Implementation Previous: Mesh Partitioning

Daniel Rypl
2005-12-03