OpenFOAM v9 User Guide

[version 12][version 11][version 10][version 9][version 8][version 7][version 6]

[chapter contents][previous topic][next topic][index]

3.4 Running applications in parallel

This section describes how to run OpenFOAM in parallel on distributed processors. The method of parallel computing used by OpenFOAM is known as domain decomposition, in which the geometry and associated ﬁelds are broken into pieces and allocated to separate processors for solution. The process of parallel computation involves: decomposition of mesh and ﬁelds; running the application in parallel; and, post-processing the decomposed case as described in the following sections. The parallel running uses the public domain openMPI implementation of the standard message passing interface (MPI) by default, although other libraries can be used.

3.4.1 Decomposition of mesh and initial ﬁeld data

The mesh and ﬁelds are decomposed using the decomposePar utility. The underlying aim is to break up the domain with minimal eﬀort but in such a way to guarantee an economic solution. The geometry and ﬁelds are broken up according to a set of parameters speciﬁed in a dictionary named decomposeParDict that must be located in the system directory of the case of interest. An example decomposeParDict dictionary is available from the interFoam/damBreak tutorial if the user requires one; the dictionary entries within it are reproduced below:

16
17numberOfSubdomains 4;
18
19method          simple;
20
21simpleCoeffs
22{
23    n               (2 2 1);
24}
25
26hierarchicalCoeffs
27{
28    n               (1 1 1);
29    order           xyz;
30}
31
32manualCoeffs
33{
34    dataFile        "";
35}
36
37distributed     no;
38
39roots           ( );
40
41
42// ************************************************************************* //

The user has a choice of four methods of decomposition, speciﬁed by the method keyword as described below.

simple: Simple geometric decomposition in which the domain is split into pieces by direction, e.g. 2 pieces in the $\text{[math]}$ direction, 1 in $\text{[math]}$ etc.
hierarchical: Hierarchical geometric decomposition which is the same as simple except the user speciﬁes the order in which the directional split is done, e.g. ﬁrst in the $\text{[math]}$ -direction, then the $\text{[math]}$ -direction etc.
scotch: Scotch decomposition which requires no geometric input from the user and attempts to minimise the number of processor boundaries. The user can specify a weighting for the decomposition between processors, through an optional processorWeights keyword which can be useful on machines with diﬀering performance between processors. There is also an optional keyword entry strategy that controls the decomposition strategy through a complex string supplied to Scotch. For more information, see the source code ﬁle: $FOAM_SRC/parallel/decompose/scotchDecomp/scotchDecomp.C
manual: Manual decomposition, where the user directly speciﬁes the allocation of each cell to a particular processor.

For each method there are a set of coeﬃcients speciﬁed in a sub-dictionary of decompositionDict, named <method>Coeﬀs as shown in the dictionary listing. The full set of keyword entries in the decomposeParDict dictionary are explained in Table 3.3 .

Compulsory entries

numberOfSubdomains	Total number of subdomains	$\text{[math]}$
method	Method of decomposition	simple / hierarchical / scotch / manual /

simpleCoeffs entries

n	Number of subdomains in $\text{[math]}$ , $\text{[math]}$ , $\text{[math]}$	( $\text{[math]}$ $\text{[math]}$ $\text{[math]}$ )
delta	Cell skew factor	Typically, $\text{[math]}$

hierarchicalCoeffs entries

n	Number of subdomains in $\text{[math]}$ , $\text{[math]}$ , $\text{[math]}$	( $\text{[math]}$ $\text{[math]}$ $\text{[math]}$ )
delta	Cell skew factor	Typically, $\text{[math]}$
order	Order of decomposition	xyz/xzy/yxz…

scotchCoeffs entries

processorWeights (optional)	List of weighting factors for allocation of cells to processors; <wt1> is the weighting factor for processor 1, etc.; weights are normalised so can take any range of values.	(<wt1>…<wtN>)
strategy	Decomposition strategy: optional and complex

manualCoeffs entries

dataFile	Name of ﬁle containing data of allocation of cells to processors	"<fileName>"

Distributed data entries (optional) — see section 3.4.4

distributed	Is the data distributed across several disks?	yes/no
roots	Root paths to case directories; <rt1> is the root path for node 1, etc.	(<rt1>…<rtN>)

Table 3.3: Keywords in decompositionDict dictionary.

The decomposePar utility is executed in the normal manner by typing

decomposePar

3.4.2 File input/output in parallel

Using standard ﬁle input/output completion, a set of subdirectories will have been created, one for each processor, in the case directory. The directories are named processor $\text{[math]}$ where $\text{[math]}$ represents a processor number and contains a time directory, containing the decomposed ﬁeld descriptions, and a constant/polyMesh directory containing the decomposed mesh description.

While this ﬁle structure is well-organised, for large parallel cases, it generates a large number of ﬁles. In very large simulations, users can experience problems including hitting limits on the number of open ﬁles imposed by the operating system.

As an alternative, the collated ﬁle format was introduced in OpenFOAM in which the data for each decomposed ﬁeld (and mesh) is collated into a single ﬁle that is written (and read) on the master processor. The ﬁles are stored in a single directory named processors.

The ﬁle writing can be threaded allowing the simulation to continue running while the data is being written to ﬁle — see below for details. NFS (Network File System) is not needed when using the collated format and, additionally, there is a masterUncollated option to write data with the original uncollated format without NFS.

The controls for the ﬁle handling are in the OptimisationSwitches of the global etc/controlDict ﬁle:

OptimisationSwitches
{
    ...

    //- Parallel IO file handler
    // uncollated (default), collated or masterUncollated
    fileHandler uncollated;

    //- collated: thread buffer size for queued file writes.
    // If set to 0 or not sufficient for the file size threading is not used.
    // Default: 2e9
    maxThreadFileBufferSize 2e9;

    //- masterUncollated: non-blocking buffer size.
    // If the file exceeds this buffer size scheduled transfer is used.
    // Default: 2e9
    maxMasterFileBufferSize 2e9;
}

3.4.2.1 Selecting the ﬁle handler

The fileHandler can be set for a speciﬁc simulation by:

over-riding the global OptimisationSwitches {fileHandler …;} in the case controlDict ﬁle;
using the -fileHandler command line argument to the solver;
setting the $FOAM_FILEHANDLER environment variable.

3.4.2.2 Updating exisiting ﬁles

A foamFormatConvert utility allows users to convert ﬁles between the collated and uncollated formats, e.g.

mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated

An example case demonstrating the ﬁle handling methods is provided in:
$FOAM_TUTORIALS/IO/ﬁleHandling

3.4.2.3 Threading support

Collated ﬁle handling runs faster with threading, especially on large cases. But it requires threading support to be enabled in the underlying MPI. Without it, the simulation will “hang” or crash. For openMPI, threading support is not set by default prior to version 2, but is generally switched on from version 2 onwards. The user can check whether openMPI is compiled with threading support by the following command:

ompi_info -c | grep -oE "MPI_THREAD_MULTIPLE[^,]*"

When using the collated ﬁle handling, memory is allocated for the data in the thread. maxThreadFileBufferSize sets the maximum size of memory that is allocated in bytes. If the data exceeds this size, the write does not use threading.

Note: if threading is not enabled in the MPI, it must be disabled for collated ﬁle handling by setting in the global etc/controlDict ﬁle:

maxThreadFileBufferSize 0;

When using the masterUncollated ﬁle handling, non-blocking MPI communication requires a suﬃciently large memory buﬀer on the master node. maxMasterFileBufferSize sets the maximum size of the buﬀer. If the data exceeds this size, the system uses scheduled communication.

3.4.3 Running a decomposed case

A decomposed OpenFOAM case is run in parallel using the openMPI implementation of MPI.

openMPI can be run on a local multiprocessor machine very simply but when running on machines across a network, a ﬁle must be created that contains the host names of the machines. The ﬁle can be given any name and located at any path. In the following description we shall refer to such a ﬁle by the generic name, including full path, <machines>.

The <machines> ﬁle contains the names of the machines listed one machine per line. The names must correspond to a fully resolved hostname in the /etc/hosts ﬁle of the machine on which the openMPI is run. The list must contain the name of the machine running the openMPI. Where a machine node contains more than one processor, the node name may be followed by the entry cpu= $\text{[math]}$ where $\text{[math]}$ is the number of processors openMPI should run on that node.

For example, let us imagine a user wishes to run openMPI from machine aaa on the following machines: aaa; bbb, which has 2 processors; and ccc. The <machines> would contain:

    aaa
    bbb cpu=2
    ccc

An application is run in parallel using mpirun.

mpirun --hostfile <machines> -np <nProcs>
<foamExec> <otherArgs> -parallel > log &

where: <nProcs> is the number of processors; <foamExec> is the executable, e.g.icoFoam; and, the output is redirected to a ﬁle named log. For example, if icoFoam is run on 4 nodes, speciﬁed in a ﬁle named machines, on the cavity tutorial in the $FOAM_RUN/tutorials/incompressible/icoFoam directory, then the following command should be executed:

mpirun --hostfile machines -np 4 icoFoam -parallel > log &

3.4.4 Distributing data across several disks

Data ﬁles may need to be distributed if, for example, if only local disks are used in order to improve performance. In this case, the user may ﬁnd that the root path to the case directory may diﬀer between machines. The paths must then be speciﬁed in the decomposeParDict dictionary using distributed and roots keywords. The distributed entry should read

distributed yes;

and the roots entry is a list of root paths, <root0>, <root1>, …, for each node

    roots
    <nRoots>
    (
       "<root0>"
       "<root1>"
       …
    );

where <nRoots> is the number of roots.

Each of the processor $\text{[math]}$ directories should be placed in the case directory at each of the root paths speciﬁed in the decomposeParDict dictionary. The system directory and ﬁles within the constant directory must also be present in each case directory. Note: the ﬁles in the constant directory are needed, but the polyMesh directory is not.

3.4.5 Post-processing parallel processed cases

When post-processing cases that have been run in parallel the user has two options:

reconstruction of the mesh and ﬁeld data to recreate the complete domain and ﬁelds, which can be post-processed as normal;
post-processing each segment of decomposed domain individually.

3.4.5.1 Reconstructing mesh and data

After a case has been run in parallel, it can be reconstructed for post-processing. The case is reconstructed by merging the sets of time directories from each processor $\text{[math]}$ directory into a single set of time directories. The reconstructPar utility performs such a reconstruction by executing the command:

reconstructPar

When the data is distributed across several disks, it must be ﬁrst copied to the local case directory for reconstruction.

3.4.5.2 Post-processing decomposed cases

The user may post-process decomposed cases using the paraFoam post-processor, described in section 6.1 . The whole simulation can be post-processed by reconstructing the case or alternatively it is possible to post-process a segment of the decomposed domain individually by simply treating the individual processor directory as a case in its own right.

OpenFOAM v9 User Guide - 3.4 Running applications in parallel