Compiling Quantum ESPRESSO-6.5.0 with Intel MPI 2018 on CentOS 7

Step 1: Download Quantum ESPRESSO 6.5.0 from Quantum ESPRESSO Download Site or git-clone QE

$ git clone https://gitlab.com/QEF/q-e.git

Step 2: Remember to source the Intel Compilers and indicate MKLROOT in your .bashrc

export MKLROOT=/usr/local/intel_2018/mkl/lib
source /usr/local/intel/2018u3/parallel_studio_xe_2018/bin/psxevars.sh intel64
source /usr/local/intel/2018u3/compilers_and_libraries/linux/bin/compilervars.sh intel64
source /usr/local/intel/2018u3/impi/2018.3.222/bin64/mpivars.sh intel64

Step 3: Make a file call setup.sh and copy the contents inside.

export F90=mpiifort
export F77=mpiifort
export MPIF90=mpiifort
export CC=mpiicc
export CPP="icc -E"
export CFLAGS=$FCFLAGS
export AR=xiar
export BLAS_LIBS=""
export LAPACK_LIBS="-lmkl_blacs_intelmpi_lp64"
export SCALAPACK_LIBS="-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
export FFT_LIBS="-L$MKLROOT/intel64"
# ./configure  --enable-parallel --prefix=/usr/local/espresso-6.5.0
# ./setup.sh
# make all -j 16 
# make install

 

Compiling OpenFOAM-5.0 with Intel-MPI

Minimum Requirements version

  1. gcc: 4.8.5
  2. cmake: 3.3 (required for ParaView and CGAL build)
  3. boost: 1.48 (required for CGAL build)
  4. fftw: 3.3.7 (optional – required for FFT-related functionality)
  5. Qt: 4.8 (optional – required for ParaView build)

I’m using Intel-16.0.4 and Intel-MPI-5.1.3.258

Step 1a: Download and Unpacking Sources

# wget -O - http://dl.openfoam.org/source/5-0 | tar xvz
# wget -O - http://dl.openfoam.org/third-party/5-0 | tar xvz

Step 1b:  Rename the Directory

# mv OpenFOAM-5.x-version-5.0 OpenFOAM-5.0
# mv ThirdParty-5.x-version-5.0 ThirdParty-5.0

Step 2: Initiate Intel and Intel-MPI Environment and source OpenFOAM-5.0 bashrc

source /usr/local/intel/bin/compilervars.sh intel64
source /usr/local/intel/parallel_studio_xe_2016.4.072/bin/psxevars.sh intel64
source /usr/local/intel/impi/5.1.3.258/bin64/mpivars.sh intel64
source /usr/local/intel/mkl/bin/mklvars.sh intel64
source /usr/local/OpenFOAM/OpenFOAM-5.0/etc/bashrc
export MPI_ROOT=/usr/local/intel/impi/5.1.3.258/intel64

Step 3: Make sure your CentOS-7 Environment have the following base packages

# yum install gcc-c++ gcc-gfortran gmp flex flex-devel boost zlib zlib-devel qt4 qt4-devel

Step 4: Edit the OpenFOAM  internal bashrc

# vim /usr/local/OpenFOAM/OpenFOAM-5.0/etc/bashrc

Line 35,36

export WM_PROJECT=OpenFOAM
export WM_PROJECT_VERSION=5.0

Line 45

FOAM_INST_DIR=/usr/local/$WM_PROJECT

Line 60

export WM_COMPILER_TYPE=system

Line 65

export WM_COMPILER=Icc

Line 88

export WM_MPLIB=INTELMPI

Step 5: Compile OpenFOAM

# ./Allwmake -update -j

Intel MPI Parameter to consider for Performance

Selection of best available communication fabrics

Suggestion 1:

 I_MPI_DEVICE I_MPI_FABRICS Description
 sock tcp  TCP/IP-enable network fabrics, such as Ethernet and Infiniband* (through IPoIB*)
 shm shm Shared-memory only
 ssm  shm:tcp  Shared-memory + TCP/IP
 rdma dapl  DAPL-capable network fabrics, such as Infiniband*, iWarp*, Dolphon*, and XPMEM* (through DAPK*)
 rdssm shm:dapl  Shared-Memory + DAPL + sockers
 ofa  OFA-capable network fabrics including Infiniband* (through OFED* verbs)
 tmi  TMI-capable network fabrics including Qlogic*, Myrinet* (through Tag Matching Interface)

 

Suggestion 2:

I_MPI_DAPL_UD Values Description
 enable
  • Connectionless feature works for DAPL fabrics only.
  • Works with OFED 1.4.2 and 2.0.24 or higher
  • Provides better scalability
  • Significant reduces memory requirements

 

Suggestion 3:

 I_MPI_PERHOST Values Remarks
 1  Make round-robin distirbution (Default value)
 all  Maps processes to all logical CPUs on a node
 allcores  Maps processes to all physical CPUs on a node

 

Suggestion 4:

I_MPI_SHM_BYPASS Values Remarks
 disable Set I_MPI_SHM_BYPASS* to ‘enable’ to turn on RDMA data exchange within single node that may outperform regular shared memory exchange. This is normally happens for large (350kb+) messages.

 

Suggestion 5:

I_MPI_ADJUST_ALLREDUCE Values Remarks
 recursive doubling algorithm 1
 Rabenseifner’s algorithm 2
 Reduce + Bcast 3
Topology aware Reduce + Bcast algorithm 4  
 Binomial gather + scatter algorithm 5
 Topology Aware Binomial Gather + scatter algorithm 6
 Ring Algorithm 7

 

Suggesion 6:

I_MPI_WAIT_MODE Values Remarks
 1 Set I_MPI_WAIT_MODE ‘to enable’ to try wait mode of the progress engine. The processes that waits for receiving that waits for receiving messages without polling of the fabrics(d) can save CPU time.

Apply wait mode to oversubscribe jobs

 

References:

  1. 23 Tips for Performance Tuning with the Intel Library (pdf)

Using Intel IMB-MPI1 to check Fabrics and expected performances

In your .bashrc, do source the

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64
MKLROOT=/usr/local/intel_2015/mkl

To simulate 3 workloads pingpong, sendrecv, and exchange with IMB-MPT1

$ mpirun -r ssh -RDMA -n 512 -env I_MPI_DEBUG 5 IMB-MPT1

 

Compiling with NWChem-6.6 with Intel MPI-5.0.3

Here is a write-up of my computing platform and applications:

  1. NWChem 6.6 (Oct 2015)
  2. Intel Compilers 2015 XE (version 15.0.6)
  3. Intel MPI (5.0.3)
  4. Intel MKL (11.2.4)
  5. Infiniband Inteconnect (OFED 1.5.3)
  6. CentOS 6.3 (x86_64)

Step 1: First thing first, source the intel components setting from

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64

Step 2: Assuming you are done, you may want to download the NWChem 6.6 from NWChem Website. You may also want to take a look at instruction set for Compiling NWChem.

I have less problem running NWCHEM when the installation and the compiling directories are the same. If you recompile, do untar from source. Somehow the “make clean” does not clean the directories properly.

# tar -zxvf Nwchem-6.6.revision27746-src.2015-10-20.tar.gz
# cd nwchem-6.6

Step 3: Apply All the Patches for the 27746 revision of NWChem 6.6

cd $NWCHEM_TOP
wget http://www.nwchem-sw.org/download.php?f=Xccvs98.patch.gz
gzip -d Xccvs98.patch.gz
patch -p0 < Xccvs98.patch

Here is my nwchem_script_Feb2017.sh. For more details information, see Compiling NWChem for details on some of the parameters

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/melvin/Downloads/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export LARGE_FILES=TRUE

export ARMCI_NETWORK=OPENIB
export IB_INCLUDE=/usr/include
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"

export MSG_COMMS=MPI
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/local/RH6_apps/intel_2015/impi_5.0.3/intel64
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpigf -lmpigi -lmpi_ilp64 -lmpi"

export FC=ifort
export CC=icc

export MKLLIB=/usr/local/RH6_apps/intel_2015/mkl/lib/intel64
export MKLINC=/usr/local/RH6_apps/intel_2015/mkl/include

export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export USE_PYTHON64=y
export PYTHONLIBTYPE=so
sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h

export HAS_BLAS=yes
export BLAS_SIZE=8 
export BLASOPT="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export LAPACK_LIBS="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export SCALAPACK_SIZE=8
export SCALAPACK="-L$MKLLIB -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export USE_64TO32=y

echo "cd $NWCHEM_TOP/src"
cd $NWCHEM_TOP/src

echo "BEGIN --- make realclean "
make realclean
echo "END --- make realclean "

echo "BEGIN --- make nwchem_config "
make nwchem_config
echo "END --- make nwchem_config "

echo "BEGIN --- make"
make CC=icc FC=ifort FOPTIMIZE=-O3 -j4
echo "END --- make "

cd $NWCHEM_TOP/src/util
make CC=icc FC=ifort FOPTIMIZE=-O3 version
make CC=icc FC=ifort FOPTIMIZE=-O3
cd $NWCHEM_TOP/src
make CC=icc FC=ifort FOPTIMIZE=-O3  link

General Site Installation

Determine the local storage path for the install files. (e.g., /usr/local/NWChem).
Make directories

# mkdir /usr/local/nwchem-6.6
# mkdir /usr/local/nwchem-6.6/bin
# mkdir /usr/local/nwchem-6.6/data

Copy binary

# cp $NWCHEM_TOP/bin/ /usr/local/nwchem-6.6/bin
# cd /usr/local/nwchem-6.6/bin
# chmod 755 nwchem

Copy libraries

# cd $NWCHEM_TOP/src/basis
# cp -r libraries /usr/local/nwchem-6.6/data

# cd $NWCHEM_TOP/src/
# cp -r data /usr/local/nwchem-6.6

# cd $NWCHEM_TOP/src/nwpw
# cp -r libraryps /usr/local/nwchem-6.6/data

The Final Lap (From Compiling NWChem)

Each user will need a .nwchemrc file to point to these default data files. A global one could be put in /usr/local/nwchem-6.6/data and a symbolic link made in each users $HOME directory is probably the best plan for new installs. Users would have to issue the following command prior to using NWChem: ln -s /usr/local/nwchem-6.6/data/default.nwchemrc $HOME/.nwchemrc

Contents of the default.nwchemrc file based on the above information should be:

nwchem_basis_library /usr/local/nwchem-6.6/data/libraries/
nwchem_nwpw_library /usr/local/nwchem-6.6/data/libraryps/
ffield amber
amber_1 /usr/local/nwchem-6.6/data/amber_s/
amber_2 /usr/local/nwchem-6.6/data/amber_q/
amber_3 /usr/local/nwchem-6.6/data/amber_x/
amber_4 /usr/local/nwchem-6.6/data/amber_u/
spce    /usr/local/nwchem-6.6/data/solvents/spce.rst
charmm_s /usr/local/nwchem-6.6/data/charmm_s/
charmm_x /usr/local/nwchem-6.6/data/charmm_x/

References:

  1. 470. Very briefly: compiling nwchem 6.3 with ifort and mkl
  2. Compiling NWChem from source
  3. Problem building NWChem version 6.5 on IB cluster with MKL & IntelMPI

Compiling VASP-5.2.12 with Intel MPI-5.0.3

Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

A. Prerequisites

To Compile VASP-5.2.12, I used

  1. Intel Compiler 15.0.6
  2. Intel MPI 5.0.3
  3. Maths Kernel Library 11.2.4

B. Compiling VASP Libraries

Assuming you have unpacked the VASP files. Here is may make file

.SUFFIXES: .inc .f .F
#-----------------------------------------------------------------------
# Makefile for LINUX NAG f90
#-----------------------------------------------------------------------
# fortran compiler
FC=ifort

# C-preprocessor
#CPP     = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C $*.F >$*.f
CPP      = gcc -E -P -C -DLONGCHAR $*.F >$*.f

CFLAGS = -O
FFLAGS = -Os -FI
FREE   = -FR

DOBJ =  preclib.o timing_.o derrf_.o dclock_.o  diolib.o dlexlib.o drdatab.o

#-----------------------------------------------------------------------
# general rules
#-----------------------------------------------------------------------

libdmy.a: $(DOBJ) linpack_double.o
-rm libdmy.a
ar vq libdmy.a $(DOBJ)

linpack_double.o: linpack_double.f
$(FC) $(FFLAGS) $(NOFREE) -c linpack_double.f

# files which do not require autodouble
lapack_double.o: lapack_double.f
$(FC) $(FFLAGS) $(NOFREE) -c lapack_double.f
lapack_single.o: lapack_single.f
$(FC) $(FFLAGS) $(NOFREE) -c lapack_single.f
#lapack_cray.o: lapack_cray.f
#       $(FC) $(FFLAGS) $(NOFREE) -c lapack_cray.f

.c.o:
$(CC) $(CFLAGS) -c $*.c
.F.o:
$(CPP)
$(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f
.F.f:
$(CPP)
.f.o:
$(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f

C. Compiling VASP

1. Copy the Makefile from makefile.linux_ifc_P4 in the vasp software.

# cp makefile.linux_ifc_P4 Makefile

2. Edit the Makefile
FC

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=mpiifort
# fortran linker
FCL=$(FC)

CPP

CPP    = $(CPP_) -DMPI  -DHOST=\"LinuxIFC\" -DIFC \
        -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \
        -DMPI_BLOCK=64000 -Duse_collective -DscaLAPACK

FFLAGS

MKLROOT=/usr/local/RH6_apps/intel_2015/mkl
MKL_PATH=$(MKLROOT)/lib/intel64
FFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw

OFLAG

#Haswell Architecture
OFLAG=-O3 -xCORE-AVX2

#Sandy-Bridge Architecture
OFLAG=-O3

The -xCORE-AVX2 is for Haswell Architecture

BLAS

BLAS= -mkl=cluster

-mkl=cluster is an Intel compiler flag that to include Intel MKL libraries, that will link with Intel MKL BLAS, LAPACK, FFT, ScaLAPACK functions that are used in VASP.

FFT3D

fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
INCS = -I$(MKLROOT)/include/fftw

LAPACK and SCALAPACK

LAPACK=
SCA=

Since the -mkl=cluster, includes MKL ScaLAPACK libraries also, it is not required to mentioned the ScaLAPACK libs. That include LAPACK

References:

  1. Building VASP* with Intel® MKL and Intel® Compilers
  2. (Intel Developer Zone)

Compiling CPMD-3.17.1 with Intel-13.0.1.117 and iMPI-14.0.2

To get the source code from CPMD, please go to http://www.cpmd.org/

Step 1: From the CPMD Directory

$ tar -zxvf cpmd-v3_17_1.tar.gz
$ cd ~/CPMD-3.17.1/CONFIGURE
$ ./mkconfig.sh IFORT-AMD64-MPI > Makefile

Step 2: Prepare the initialization

source /usr/local/intel_2013sp1/composerxe/mkl/bin/mklvars.sh intel64
source /usr/local/intel_2013sp1/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2013sp1/impi/4.1.3.048/intel64/bin/mpivars.sh intel64
source /usr/local/intel_2013sp1/composerxe/tbb/bin/tbbvars.sh intel64
source /usr/local/intel_2013sp1/itac/8.1.4.045/intel64/bin/itacvars.sh

Step 3:  I’m using MKL for the Mathematical Libraries

#--------------- Default Configuration for IFORT-AMD64-MPI ---------------
SRC  = .
DEST = .
BIN  = .
MKLPATH = /usr/local/intel_2013sp1/mkl/lib/intel64
MKLINCLUDE = /usr/local/intel_2013sp1/mkl/include
FFLAGS = -I/usr/local/intel_2013sp1/impi/4.1.3.048/include64 -L/usr/local/intel_2013sp1/impi/4.1.3.048/lib64
LFLAGS = -L${MKLPATH} -I${MKLINCLUDE} -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
CFLAGS =
CPP = /lib/cpp -P -C -traditional
CPPFLAGS = -DPOINTER8 -DFFT_DEFAULT -DPARALLEL=parallel -DMAIA-x86_64-INTEL-IMPI -DINTEL_MKL -D__Linux
NOOPT_FLAG =
CC = mpicc
FC = mpiifort -c
LD = mpiifort -static-intel
AR = ar
#----------------------------------------------------------------------------

Step 4: Compile CPMD

$ make

If the compilation succeed, it should generate a cpmd.x executable.

Step 5: Pathing
Make sure your $PATH reflect the path of the executable cpmd.x. It is also important to ensure that you check that the libraries are properly linked to the executable

# ldd cpmd.x