Parallelising Codes 1

Distributed Memory Parallelism

With distributed memory parallelism, the computer is treated as though it were a set of completely separate processors each with their own separate memory, and that these computers are then connected together with a high speed network. This type of parallelism is equally valid on real distributed systems, such as workstation clusters, and shared memory machines such as the CSAR Origin and Altix machines.

Programs written to take advantage of distributed memory parallelism are often the best parallel codes in terms of scalability with some codes able to achieve good scaling up to thousands of processors.

The most common way of coding programs to use distributed memory parallelism is to use a message passing library, and the most important and widely used of these is MPI. The MPI library provides a way of transferring data between communicating processes so that they can solve large problems in a short time by having each process tackle a smaller part of the whole. Note that for efficient use of the MPI library the number of processors available should be the same as the number of processes which are launched to run the program.

Distributed memory codes can be written in different ways depending upon the type of work to be done, for example there could be some form of task farming where one process is used simply to control the work which the other processes are given, or all processes could be working together on a large problem. MPI codes may be written so that different processes actually run different executables, but most developers write their software so that all processes run instances of the same program even where they then run completely different chunks of the code. This paradigm, where all processes use the same code to work on distributed data is called SIMD which stands for Single Instruction Multiple Data.

When programming in a distributed memory fashion the entire code needs to be parallelised in one go since the data is physically distributed among the processors. For this reason, modifying serial codes to become distributed memory codes is a difficult and time consuming operation which has the possibility to introduce numerous bugs.

If you intend to parallelise your code in a distributed data fashion with MPI then you should look at the communication libraries pages in the tools section of this web site.