Compiling OpenFOAM-9 with Third-Party-9 with Intel MPI on CentOS 7

Step 1a: Get the Software

If you do have root access to the machine, the recommended installation directory is $HOME/OpenFOAM . If you have root permissions and the installation is for more than one user, one of the ‘standard’ locations can be used, e.g. /usr/local/OpenFOAM.

# wget -O - http://dl.openfoam.org/source/9 | tar xvz
# wget -O - http://dl.openfoam.org/third-party/9 | tar xvz

Step 1b: The files unpack to produce directories OpenFOAM-9-version-9 and ThirdParty-9-version-9, which need to be renamed as follows:

# mv OpenFOAM-9-version-9 OpenFOAM-9
# mv ThirdParty-9-version-9 ThirdParty-9

Step 2a: Load Intel Compilers. I loaded the Intel Parallel Cluster Suite 2018

# source /usr/local/intel/2018u3/bin/compilervars.sh intel64
# source /usr/local/intel/2018u3/mkl/bin/mklvars.sh intel64
# source /usr/local/intel/2018u3/impi/2018.3.222/bin64/mpivars.sh intel64
# source /usr/local/intel/2018u3/parallel_studio_xe_2018/bin/psxevars.sh intel64
# export MPI_ROOT=/usr/local/intel/2018u3/impi/2018.3.222/intel64

Step 2b: Create softlink for include64 and lib64 for Intel MPI (if requried only). You can check first. If it is there, it should be something like this.

# ls -l /usr/local/intel/2018u3/impi/2018.3.222/intel64/include64
# lrwxrwxrwx 1 root hpccentrifyusers 7 Aug  9  2019 /usr/local/intel/2018u3/impi/2018.3.222/intel64/include64 -> include

If not there, you may want to do this.

# cd /usr/local/intel/2018u3/impi/2018.3.222/intel64
# ln -s include include64
# ln -s lib lib64

Step 3: Edit the OpenFOAM bashrc

# vim /usr/local/OpenFOAM/OpenFOAM-9/etc/bashrc
.....
export FOAM_INST_DIR=$(cd $(dirname ${BASH_SOURCE:-$0})/../.. && pwd -P) || \
export FOAM_INST_DIR=/usr/local/$WM_PROJECT
.....
#- Compiler location:
#    WM_COMPILER_TYPE= system | ThirdParty (OpenFOAM)
export WM_COMPILER_TYPE=system

#- Compiler:
#    WM_COMPILER = Gcc | Gcc48 ... Gcc62 | Clang | Icc
export WM_COMPILER=Icc
unset WM_COMPILER_ARCH WM_COMPILER_LIB_ARCH

#- Memory addressing:
#    On a 64bit OS this can be 32bit or 64bit
#    On a 32bit OS addressing is 32bit and this option is not used
#    WM_ARCH_OPTION = 32 | 64
export WM_ARCH_OPTION=64

#- Precision:
#    WM_PRECISION_OPTION = SP | DP | LP
export WM_PRECISION_OPTION=DP

#- Label size:
#    WM_LABEL_SIZE = 32 | 64
export WM_LABEL_SIZE=32

#- Optimised, debug, profiling:
#    WM_COMPILE_OPTION = Opt | Debug | Prof
export WM_COMPILE_OPTION=Opt

#- MPI implementation:
#    WM_MPLIB = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPICH | MPICH-GM | HPMPI
#               | MPI | FJMPI | QSMPI | SGIMPI | INTELMPI
export WM_MPLIB=INTELMPI

#- Operating System:
#    WM_OSTYPE = POSIX | ???
export WM_OSTYPE=POSIX
.....

Step 4a: Edit the ThirdParty-9 scotch_6.0.9 Packages

$ cd /usr/local/OpenFOAM/ThirdParty-9/scotch_6.0.9/src

Step 4b: Copy the right Architecture and place it with the directory that holds the Makefile. For example we wanted “Makefile.inc.x86-64_pc_linux2.icc.impi”

# cp /usr/local/OpenFOAM/ThirdParty-9/scotch_6.0.9/src/Make.inc/Makefile.inc.x86-64_pc_linux2.icc.impi /usr/local/OpenFOAM/ThirdParty-9/scotch_6.0.9/src

Step 5: Go back to OpenFOAM source directory

# source /usr/local/OpenFOAM/OpenFOAM-9/etc/bashrc
# cd /usr/local/OpenFOAM/OpenFOAM-9
# ./Allwmake -j 16 | tee Allwmake.log