Compiling OpenFOAM-9 with Third-Party-9 with Intel MPI on CentOS 7

Step 1a: Get the Software

If you do have root access to the machine, the recommended installation directory is $HOME/OpenFOAM . If you have root permissions and the installation is for more than one user, one of the ‘standard’ locations can be used, e.g. /usr/local/OpenFOAM.

# wget -O - http://dl.openfoam.org/source/9 | tar xvz
# wget -O - http://dl.openfoam.org/third-party/9 | tar xvz

Step 1b: The files unpack to produce directories OpenFOAM-9-version-9 and ThirdParty-9-version-9, which need to be renamed as follows:

# mv OpenFOAM-9-version-9 OpenFOAM-9
# mv ThirdParty-9-version-9 ThirdParty-9

Step 2a: Load Intel Compilers. I loaded the Intel Parallel Cluster Suite 2018

# source /usr/local/intel/2018u3/bin/compilervars.sh intel64
# source /usr/local/intel/2018u3/mkl/bin/mklvars.sh intel64
# source /usr/local/intel/2018u3/impi/2018.3.222/bin64/mpivars.sh intel64
# source /usr/local/intel/2018u3/parallel_studio_xe_2018/bin/psxevars.sh intel64
# export MPI_ROOT=/usr/local/intel/2018u3/impi/2018.3.222/intel64

Step 2b: Create softlink for include64 and lib64 for Intel MPI (if requried only). You can check first. If it is there, it should be something like this.

# ls -l /usr/local/intel/2018u3/impi/2018.3.222/intel64/include64
# lrwxrwxrwx 1 root hpccentrifyusers 7 Aug  9  2019 /usr/local/intel/2018u3/impi/2018.3.222/intel64/include64 -> include

If not there, you may want to do this.

# cd /usr/local/intel/2018u3/impi/2018.3.222/intel64
# ln -s include include64
# ln -s lib lib64

Step 3: Edit the OpenFOAM bashrc

# vim /usr/local/OpenFOAM/OpenFOAM-9/etc/bashrc
.....
export FOAM_INST_DIR=$(cd $(dirname ${BASH_SOURCE:-$0})/../.. && pwd -P) || \
export FOAM_INST_DIR=/usr/local/$WM_PROJECT
.....
#- Compiler location:
#    WM_COMPILER_TYPE= system | ThirdParty (OpenFOAM)
export WM_COMPILER_TYPE=system

#- Compiler:
#    WM_COMPILER = Gcc | Gcc48 ... Gcc62 | Clang | Icc
export WM_COMPILER=Icc
unset WM_COMPILER_ARCH WM_COMPILER_LIB_ARCH

#- Memory addressing:
#    On a 64bit OS this can be 32bit or 64bit
#    On a 32bit OS addressing is 32bit and this option is not used
#    WM_ARCH_OPTION = 32 | 64
export WM_ARCH_OPTION=64

#- Precision:
#    WM_PRECISION_OPTION = SP | DP | LP
export WM_PRECISION_OPTION=DP

#- Label size:
#    WM_LABEL_SIZE = 32 | 64
export WM_LABEL_SIZE=32

#- Optimised, debug, profiling:
#    WM_COMPILE_OPTION = Opt | Debug | Prof
export WM_COMPILE_OPTION=Opt

#- MPI implementation:
#    WM_MPLIB = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPICH | MPICH-GM | HPMPI
#               | MPI | FJMPI | QSMPI | SGIMPI | INTELMPI
export WM_MPLIB=INTELMPI

#- Operating System:
#    WM_OSTYPE = POSIX | ???
export WM_OSTYPE=POSIX
.....

Step 4a: Edit the ThirdParty-9 scotch_6.0.9 Packages

$ cd /usr/local/OpenFOAM/ThirdParty-9/scotch_6.0.9/src

Step 4b: Copy the right Architecture and place it with the directory that holds the Makefile. For example we wanted “Makefile.inc.x86-64_pc_linux2.icc.impi”

# cp /usr/local/OpenFOAM/ThirdParty-9/scotch_6.0.9/src/Make.inc/Makefile.inc.x86-64_pc_linux2.icc.impi /usr/local/OpenFOAM/ThirdParty-9/scotch_6.0.9/src

Step 5: Go back to OpenFOAM source directory

# source /usr/local/OpenFOAM/OpenFOAM-9/etc/bashrc
# cd /usr/local/OpenFOAM/OpenFOAM-9
# ./Allwmake -j 16 | tee Allwmake.log

Compiling OpenFOAM-7 with Third-Party-7 with Intel MPI on CentOS 7

Step 1: Create a Directory git clone OpenFOAM-7 and Third-Party-7

# mkdir -p /usr/local/OpenFOAM
# cd OpenFOAM
# git clone https://github.com/OpenFOAM/OpenFOAM-7.git
# git clone https://github.com/OpenFOAM/ThirdParty-7.git

Step 2a: Load Intel Compilers. I loaded the Intel Parallel Cluster Suite 2018

# source /usr/local/intel/2018u3/bin/compilervars.sh intel64
# source /usr/local/intel/2018u3/mkl/bin/mklvars.sh intel64
# source /usr/local/intel/2018u3/impi/2018.3.222/bin64/mpivars.sh intel64
# source /usr/local/intel/2018u3/parallel_studio_xe_2018/bin/psxevars.sh intel64
# export MPI_ROOT=/usr/local/intel/2018u3/impi/2018.3.222/intel64

Step 2b: Create softlink for include64 and lib64 for Intel MPI (if requried only). You can check first. If it is there, it should be something like this.

# ls -l /usr/local/intel/2018u3/impi/2018.3.222/intel64/include64
# lrwxrwxrwx 1 root hpccentrifyusers 7 Aug  9  2019 /usr/local/intel/2018u3/impi/2018.3.222/intel64/include64 -> include

If not there, you may want to do this.

# cd /usr/local/intel/2018u3/impi/2018.3.222/intel64
# ln -s include include64
# ln -s lib lib64

Step 3: Edit the OpenFOAM bashrc

# vim /usr/local/OpenFOAM/OpenFOAM-7/etc/bashrc
......
export FOAM_INST_DIR=/usr/local/$WM_PROJECT
.....
#- Compiler:
# WM_COMPILER = Gcc | Gcc48 ... Gcc62 | Clang | Icc
export WM_COMPILER=Icc
.....
#- MPI implementation:
# WM_MPLIB = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPICH | MPICH-GM | HPMPI
# | MPI | FJMPI | QSMPI | SGIMPI | INTELMPI
export WM_MPLIB=INTELMPI
.....

Step 4: Edit the ThirdParty-7 scotch_6.0.6 Packages

$ cd /usr/local/OpenFOAM/ThirdParty-7/scotch_6.0.9/src
$ vim /usr/local/OpenFOAM/ThirdParty-7/scotch_6.0.9/src/Makefile.inc
AR = icc
...
CCS = icc
CCP = mpicc
...
CFLAGS = $(WM_CFLAGS) -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -
DSCOTCH_RENAME -Drestrict=__restrict -I$(MPI_ROOT)/include64 -L$(MPI_ROOT)/lib64

Step 5: Go back to OpenFOAM source directory

# source /usr/local/OpenFOAM/OpenFOAM-7/etc/bashrc
# cd /usr/local/OpenFOAM/OpenFOAM-7
# ./Allwmake -j 16 | tee Allwmake.log

Step 6: Testing the OpenFOAM

# ./usr/local/OpenFOAM/OpenFOAM-7/bin/foamInstallationTest