Using Intel IMB-MPI1 to check Fabrics and expected performances

In your .bashrc, do source the

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64
MKLROOT=/usr/local/intel_2015/mkl

To simulate 3 workloads pingpong, sendrecv, and exchange with IMB-MPT1

$ mpirun -r ssh -RDMA -n 512 -env I_MPI_DEBUG 5 IMB-MPT1

 

Compiling Intel FFTW3 and FFTW2 Interface Wrapper Library

FFTW3 wrappers to Intel MKL are delivered both in Intel MKL and as source code which can be compiled to build to build standalone wrapper library with exactly the same functionality.

The source code for the wrappers, makefiles are found …..\interfaces\fftw3xc subdirectory in the Intel MKL Directory

Intel FFTW3 Interface Wrapper Library. Do the same for fftw3xc and fftw3xf

# cd $MKLROOT
# cd interfaces/fftw3xc
# make libintel64  INSTALL_DIR=$MKLROOT/lib/intel64
# cd $MKLROOT
# cd interfaces/fftw3xf
# make libintel64  INSTALL_DIR=$MKLROOT/lib/intel64

Once Compiled, the libraries are kept $MKLROOT/lib/intel64

Intel FFTW2 Interface Wrapper Library. Do the same for fftw2xc and fftw2xf

# cd $MKLROOT
# cd interfaces/fftw2xc
# make libintel64  PRECISION=MKL_DOUBLE
# make libintel64  PRECISION=MKL_SINGLE
# cd $MKLROOT
# cd interfaces/fftw2xf
# make libintel64  PRECISION=MKL_DOUBLE INSTALL_DIR=$MKLROOT/lib/intel64
# make libintel64  PRECISION=MKL_SINGLE INSTALL_DIR=$MKLROOT/lib/intel64

Once Compiled, the libraries are kept $MKLROOT/lib/intel64

Compiling with NWChem-6.6 with Intel MPI-5.0.3

Here is a write-up of my computing platform and applications:

  1. NWChem 6.6 (Oct 2015)
  2. Intel Compilers 2015 XE (version 15.0.6)
  3. Intel MPI (5.0.3)
  4. Intel MKL (11.2.4)
  5. Infiniband Inteconnect (OFED 1.5.3)
  6. CentOS 6.3 (x86_64)

Step 1: First thing first, source the intel components setting from

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64

Step 2: Assuming you are done, you may want to download the NWChem 6.6 from NWChem Website. You may also want to take a look at instruction set for Compiling NWChem.

I have less problem running NWCHEM when the installation and the compiling directories are the same. If you recompile, do untar from source. Somehow the “make clean” does not clean the directories properly.

# tar -zxvf Nwchem-6.6.revision27746-src.2015-10-20.tar.gz
# cd nwchem-6.6

Step 3: Apply All the Patches for the 27746 revision of NWChem 6.6

cd $NWCHEM_TOP
wget http://www.nwchem-sw.org/download.php?f=Xccvs98.patch.gz
gzip -d Xccvs98.patch.gz
patch -p0 < Xccvs98.patch

Here is my nwchem_script_Feb2017.sh. For more details information, see Compiling NWChem for details on some of the parameters

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/melvin/Downloads/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export LARGE_FILES=TRUE

export ARMCI_NETWORK=OPENIB
export IB_INCLUDE=/usr/include
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"

export MSG_COMMS=MPI
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/local/RH6_apps/intel_2015/impi_5.0.3/intel64
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpigf -lmpigi -lmpi_ilp64 -lmpi"

export FC=ifort
export CC=icc

export MKLLIB=/usr/local/RH6_apps/intel_2015/mkl/lib/intel64
export MKLINC=/usr/local/RH6_apps/intel_2015/mkl/include

export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export USE_PYTHON64=y
export PYTHONLIBTYPE=so
sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h

export HAS_BLAS=yes
export BLAS_SIZE=8 
export BLASOPT="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export LAPACK_LIBS="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export SCALAPACK_SIZE=8
export SCALAPACK="-L$MKLLIB -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export USE_64TO32=y

echo "cd $NWCHEM_TOP/src"
cd $NWCHEM_TOP/src

echo "BEGIN --- make realclean "
make realclean
echo "END --- make realclean "

echo "BEGIN --- make nwchem_config "
make nwchem_config
echo "END --- make nwchem_config "

echo "BEGIN --- make"
make CC=icc FC=ifort FOPTIMIZE=-O3 -j4
echo "END --- make "

cd $NWCHEM_TOP/src/util
make CC=icc FC=ifort FOPTIMIZE=-O3 version
make CC=icc FC=ifort FOPTIMIZE=-O3
cd $NWCHEM_TOP/src
make CC=icc FC=ifort FOPTIMIZE=-O3  link

General Site Installation

Determine the local storage path for the install files. (e.g., /usr/local/NWChem).
Make directories

# mkdir /usr/local/nwchem-6.6
# mkdir /usr/local/nwchem-6.6/bin
# mkdir /usr/local/nwchem-6.6/data

Copy binary

# cp $NWCHEM_TOP/bin/ /usr/local/nwchem-6.6/bin
# cd /usr/local/nwchem-6.6/bin
# chmod 755 nwchem

Copy libraries

# cd $NWCHEM_TOP/src/basis
# cp -r libraries /usr/local/nwchem-6.6/data

# cd $NWCHEM_TOP/src/
# cp -r data /usr/local/nwchem-6.6

# cd $NWCHEM_TOP/src/nwpw
# cp -r libraryps /usr/local/nwchem-6.6/data

The Final Lap (From Compiling NWChem)

Each user will need a .nwchemrc file to point to these default data files. A global one could be put in /usr/local/nwchem-6.6/data and a symbolic link made in each users $HOME directory is probably the best plan for new installs. Users would have to issue the following command prior to using NWChem: ln -s /usr/local/nwchem-6.6/data/default.nwchemrc $HOME/.nwchemrc

Contents of the default.nwchemrc file based on the above information should be:

nwchem_basis_library /usr/local/nwchem-6.6/data/libraries/
nwchem_nwpw_library /usr/local/nwchem-6.6/data/libraryps/
ffield amber
amber_1 /usr/local/nwchem-6.6/data/amber_s/
amber_2 /usr/local/nwchem-6.6/data/amber_q/
amber_3 /usr/local/nwchem-6.6/data/amber_x/
amber_4 /usr/local/nwchem-6.6/data/amber_u/
spce    /usr/local/nwchem-6.6/data/solvents/spce.rst
charmm_s /usr/local/nwchem-6.6/data/charmm_s/
charmm_x /usr/local/nwchem-6.6/data/charmm_x/

References:

  1. 470. Very briefly: compiling nwchem 6.3 with ifort and mkl
  2. Compiling NWChem from source
  3. Problem building NWChem version 6.5 on IB cluster with MKL & IntelMPI

Compiling VASP-5.2.12 with Intel MPI-5.0.3

Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

A. Prerequisites

To Compile VASP-5.2.12, I used

  1. Intel Compiler 15.0.6
  2. Intel MPI 5.0.3
  3. Maths Kernel Library 11.2.4

B. Compiling VASP Libraries

Assuming you have unpacked the VASP files. Here is may make file

.SUFFIXES: .inc .f .F
#-----------------------------------------------------------------------
# Makefile for LINUX NAG f90
#-----------------------------------------------------------------------
# fortran compiler
FC=ifort

# C-preprocessor
#CPP     = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C $*.F >$*.f
CPP      = gcc -E -P -C -DLONGCHAR $*.F >$*.f

CFLAGS = -O
FFLAGS = -Os -FI
FREE   = -FR

DOBJ =  preclib.o timing_.o derrf_.o dclock_.o  diolib.o dlexlib.o drdatab.o

#-----------------------------------------------------------------------
# general rules
#-----------------------------------------------------------------------

libdmy.a: $(DOBJ) linpack_double.o
-rm libdmy.a
ar vq libdmy.a $(DOBJ)

linpack_double.o: linpack_double.f
$(FC) $(FFLAGS) $(NOFREE) -c linpack_double.f

# files which do not require autodouble
lapack_double.o: lapack_double.f
$(FC) $(FFLAGS) $(NOFREE) -c lapack_double.f
lapack_single.o: lapack_single.f
$(FC) $(FFLAGS) $(NOFREE) -c lapack_single.f
#lapack_cray.o: lapack_cray.f
#       $(FC) $(FFLAGS) $(NOFREE) -c lapack_cray.f

.c.o:
$(CC) $(CFLAGS) -c $*.c
.F.o:
$(CPP)
$(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f
.F.f:
$(CPP)
.f.o:
$(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f

C. Compiling VASP

1. Copy the Makefile from makefile.linux_ifc_P4 in the vasp software.

# cp makefile.linux_ifc_P4 Makefile

2. Edit the Makefile
FC

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=mpiifort
# fortran linker
FCL=$(FC)

CPP

CPP    = $(CPP_) -DMPI  -DHOST=\"LinuxIFC\" -DIFC \
        -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \
        -DMPI_BLOCK=64000 -Duse_collective -DscaLAPACK

FFLAGS

MKLROOT=/usr/local/RH6_apps/intel_2015/mkl
MKL_PATH=$(MKLROOT)/lib/intel64
FFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw

OFLAG

#Haswell Architecture
OFLAG=-O3 -xCORE-AVX2

#Sandy-Bridge Architecture
OFLAG=-O3

The -xCORE-AVX2 is for Haswell Architecture

BLAS

BLAS= -mkl=cluster

-mkl=cluster is an Intel compiler flag that to include Intel MKL libraries, that will link with Intel MKL BLAS, LAPACK, FFT, ScaLAPACK functions that are used in VASP.

FFT3D

fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
INCS = -I$(MKLROOT)/include/fftw

LAPACK and SCALAPACK

LAPACK=
SCA=

Since the -mkl=cluster, includes MKL ScaLAPACK libraries also, it is not required to mentioned the ScaLAPACK libs. That include LAPACK

References:

  1. Building VASP* with Intel® MKL and Intel® Compilers
  2. (Intel Developer Zone)

Compiling CPMD-3.17.1 with Intel-13.0.1.117 and iMPI-14.0.2

To get the source code from CPMD, please go to http://www.cpmd.org/

Step 1: From the CPMD Directory

$ tar -zxvf cpmd-v3_17_1.tar.gz
$ cd ~/CPMD-3.17.1/CONFIGURE
$ ./mkconfig.sh IFORT-AMD64-MPI > Makefile

Step 2: Prepare the initialization

source /usr/local/intel_2013sp1/composerxe/mkl/bin/mklvars.sh intel64
source /usr/local/intel_2013sp1/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2013sp1/impi/4.1.3.048/intel64/bin/mpivars.sh intel64
source /usr/local/intel_2013sp1/composerxe/tbb/bin/tbbvars.sh intel64
source /usr/local/intel_2013sp1/itac/8.1.4.045/intel64/bin/itacvars.sh

Step 3:  I’m using MKL for the Mathematical Libraries

#--------------- Default Configuration for IFORT-AMD64-MPI ---------------
SRC  = .
DEST = .
BIN  = .
MKLPATH = /usr/local/intel_2013sp1/mkl/lib/intel64
MKLINCLUDE = /usr/local/intel_2013sp1/mkl/include
FFLAGS = -I/usr/local/intel_2013sp1/impi/4.1.3.048/include64 -L/usr/local/intel_2013sp1/impi/4.1.3.048/lib64
LFLAGS = -L${MKLPATH} -I${MKLINCLUDE} -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
CFLAGS =
CPP = /lib/cpp -P -C -traditional
CPPFLAGS = -DPOINTER8 -DFFT_DEFAULT -DPARALLEL=parallel -DMAIA-x86_64-INTEL-IMPI -DINTEL_MKL -D__Linux
NOOPT_FLAG =
CC = mpicc
FC = mpiifort -c
LD = mpiifort -static-intel
AR = ar
#----------------------------------------------------------------------------

Step 4: Compile CPMD

$ make

If the compilation succeed, it should generate a cpmd.x executable.

Step 5: Pathing
Make sure your $PATH reflect the path of the executable cpmd.x. It is also important to ensure that you check that the libraries are properly linked to the executable

# ldd cpmd.x

List of mkl_solver* libraries are deprecated libraries since version 10.2 Update 2

Taken from mkl_solver* libraries are deprecated libraries since version 10.2 Update 2

Since version 10.2 update 2 of Intel® MKL,

all components of Direct Solver (Pardiso and DSS), Trust-Region (TR) Solver, Iterative Sparse Solver (ISS) and GNU Multiple Precision (GMP) were moved to standard MKL libraries.

So now solver ( e.g: mkl_solver.lib and mkl_solver_sequential.lib for IA32 ) libraries are

empty (for backward compatibility).

The list of deprecated libraries are the following:

Intel® MKL for Linux:

lib/32/libmkl_solver.a
lib/32/libmkl_solver_sequential.a
lib/em64t/libmkl_solver_ilp64.a
lib/em64t/libmkl_solver_ilp64_sequential.a
lib/em64t/libmkl_solver_lp64.a
lib/em64t/libmkl_solver_lp64_sequential.a
lib/ia64/libmkl_solver_ilp64.a
lib/ia64/libmkl_solver_ilp64_sequential.a
lib/ia64/libmkl_solver_lp64.a
lib/ia64/libmkl_solver_lp64_sequential.a

Therefore, the updated linking line will look like:

Linking on Intel®64:

static linking:

ifort pardiso.f -L$MKLPATH -I$MKLINCLUDE \
-Wl,–start-group \
$MKLPATH/libmkl_intel_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a \
-Wl,–end-group -liomp5 -lpthread

dynamic linking:

ifort pardiso.f -L$MKLPATH -I$MKLINCLUDE \
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread

where:
In these examples,

MKLPATH=$MKLROOT/lib/em64t
MKLINCLUDE=$MKLROOT/include.

Compiling OpenMPI 1.6.5 with Intel 12.1.5 on CentOS 6

Step 1: Download the OpenMPI Software from http://www.open-mpi.org/ . The current stable version at point of writing is OpenMPI 1.6.5

Step 2: Download and Install the Intel Compilers from Intel Website. More information can be taken from Free Non-Commercial Intel Compiler Download

Step 3: Add the Intel Directory Binary Path to the Bash Startup

At my ~/.bash_profile directory, I've added
source /usr/local/intel/composerxe/bin/compilervars.sh intel64
export PATH=$PATH:/usr/local/intel/composerxe/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/RH6_apps/intel/composerxe/lib/intel64:/usr/local/intel/composerxe/mkl/lib/intel64

At command prompt

# source .bashrc

Step 4: Configuration Information

# gunzip -c openmpi-1.6.5.tar.gz tar xf -
# cd openmpi-1.6.5
# /configure --prefix=/usr/local/openmpi-1.6.5-intel-v12.1.5 CC=icc CXX=icpc F77=ifort FC=ifort --with-devel-headers --enable-binaries
# make all install

You will need to include –devel-headers –enable-binaries” or you will face the error Cannot open OpenMPI Configuration mpif90-wrapper-data.txt

Step 5: Test the configuration

$ mpicc -v
cc version 12.1.5 (gcc version 4.4.6 compatibility)