Author: kittycool only

by

Compiling Intel FFTW3 and FFTW2 Interface Wrapper Library

FFTW3 wrappers to Intel MKL are delivered both in Intel MKL and as source code which can be compiled to build to build standalone wrapper library with exactly the same functionality.

The source code for the wrappers, makefiles are found …..\interfaces\fftw3xc subdirectory in the Intel MKL Directory

Intel FFTW3 Interface Wrapper Library. Do the same for fftw3xc and fftw3xf

# cd $MKLROOT
# cd interfaces/fftw3xc
# make libintel64  INSTALL_DIR=$MKLROOT/lib/intel64
# cd $MKLROOT
# cd interfaces/fftw3xf
# make libintel64  INSTALL_DIR=$MKLROOT/lib/intel64

Once Compiled, the libraries are kept $MKLROOT/lib/intel64

Intel FFTW2 Interface Wrapper Library. Do the same for fftw2xc and fftw2xf

# cd $MKLROOT
# cd interfaces/fftw2xc
# make libintel64  PRECISION=MKL_DOUBLE
# make libintel64  PRECISION=MKL_SINGLE
# cd $MKLROOT
# cd interfaces/fftw2xf
# make libintel64  PRECISION=MKL_DOUBLE INSTALL_DIR=$MKLROOT/lib/intel64
# make libintel64  PRECISION=MKL_SINGLE INSTALL_DIR=$MKLROOT/lib/intel64

Once Compiled, the libraries are kept $MKLROOT/lib/intel64

by

Compiling with NWChem-6.6 with Intel MPI-5.0.3

Here is a write-up of my computing platform and applications:

  1. NWChem 6.6 (Oct 2015)
  2. Intel Compilers 2015 XE (version 15.0.6)
  3. Intel MPI (5.0.3)
  4. Intel MKL (11.2.4)
  5. Infiniband Inteconnect (OFED 1.5.3)
  6. CentOS 6.3 (x86_64)

Step 1: First thing first, source the intel components setting from

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64

Step 2: Assuming you are done, you may want to download the NWChem 6.6 from NWChem Website. You may also want to take a look at instruction set for Compiling NWChem.

I have less problem running NWCHEM when the installation and the compiling directories are the same. If you recompile, do untar from source. Somehow the “make clean” does not clean the directories properly.

# tar -zxvf Nwchem-6.6.revision27746-src.2015-10-20.tar.gz
# cd nwchem-6.6

Step 3: Apply All the Patches for the 27746 revision of NWChem 6.6 

cd $NWCHEM_TOP
wget http://www.nwchem-sw.org/download.php?f=Xccvs98.patch.gz
gzip -d Xccvs98.patch.gz
patch -p0 < Xccvs98.patch

Here is my nwchem_script_Feb2017.sh. For more details information, see Compiling NWChem for details on some of the parameters

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/melvin/Downloads/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export LARGE_FILES=TRUE

export ARMCI_NETWORK=OPENIB
export IB_INCLUDE=/usr/include
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"

export MSG_COMMS=MPI
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/local/RH6_apps/intel_2015/impi_5.0.3/intel64
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpigf -lmpigi -lmpi_ilp64 -lmpi"

export FC=ifort
export CC=icc

export MKLLIB=/usr/local/RH6_apps/intel_2015/mkl/lib/intel64
export MKLINC=/usr/local/RH6_apps/intel_2015/mkl/include

export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export USE_PYTHON64=y
export PYTHONLIBTYPE=so
sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h

export HAS_BLAS=yes
export BLAS_SIZE=8 
export BLASOPT="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export LAPACK_LIBS="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export SCALAPACK_SIZE=8
export SCALAPACK="-L$MKLLIB -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export USE_64TO32=y

echo "cd $NWCHEM_TOP/src"
cd $NWCHEM_TOP/src

echo "BEGIN --- make realclean "
make realclean
echo "END --- make realclean "

echo "BEGIN --- make nwchem_config "
make nwchem_config
echo "END --- make nwchem_config "

echo "BEGIN --- make"
make CC=icc FC=ifort FOPTIMIZE=-O3 -j4
echo "END --- make "

cd $NWCHEM_TOP/src/util
make CC=icc FC=ifort FOPTIMIZE=-O3 version
make CC=icc FC=ifort FOPTIMIZE=-O3
cd $NWCHEM_TOP/src
make CC=icc FC=ifort FOPTIMIZE=-O3  link

General Site Installation

Determine the local storage path for the install files. (e.g., /usr/local/NWChem).
Make directories

# mkdir /usr/local/nwchem-6.6
# mkdir /usr/local/nwchem-6.6/bin
# mkdir /usr/local/nwchem-6.6/data

Copy binary

# cp $NWCHEM_TOP/bin/ /usr/local/nwchem-6.6/bin
# cd /usr/local/nwchem-6.6/bin
# chmod 755 nwchem

Copy libraries

# cd $NWCHEM_TOP/src/basis
# cp -r libraries /usr/local/nwchem-6.6/data

# cd $NWCHEM_TOP/src/
# cp -r data /usr/local/nwchem-6.6

# cd $NWCHEM_TOP/src/nwpw
# cp -r libraryps /usr/local/nwchem-6.6/data

The Final Lap (From Compiling NWChem)

Each user will need a .nwchemrc file to point to these default data files. A global one could be put in /usr/local/nwchem-6.6/data and a symbolic link made in each users $HOME directory is probably the best plan for new installs. Users would have to issue the following command prior to using NWChem: ln -s /usr/local/nwchem-6.6/data/default.nwchemrc $HOME/.nwchemrc

Contents of the default.nwchemrc file based on the above information should be:

nwchem_basis_library /usr/local/nwchem-6.6/data/libraries/
nwchem_nwpw_library /usr/local/nwchem-6.6/data/libraryps/
ffield amber
amber_1 /usr/local/nwchem-6.6/data/amber_s/
amber_2 /usr/local/nwchem-6.6/data/amber_q/
amber_3 /usr/local/nwchem-6.6/data/amber_x/
amber_4 /usr/local/nwchem-6.6/data/amber_u/
spce    /usr/local/nwchem-6.6/data/solvents/spce.rst
charmm_s /usr/local/nwchem-6.6/data/charmm_s/
charmm_x /usr/local/nwchem-6.6/data/charmm_x/

References:

  1. 470. Very briefly: compiling nwchem 6.3 with ifort and mkl
  2. Compiling NWChem from source
  3. Problem building NWChem version 6.5 on IB cluster with MKL & IntelMPI
by

Compiling swig-3.0.12 on CentOS 6

Compile swig-3.0.12

Do download http://www.swig.org/index.php.
PCRE needs to be installed on your system to build SWIG, in particular pcre-config must be available. For more information on SWIG installation do take a look at http://www.swig.org/Doc3.0/SWIGDocumentation.html

./configure --prefix=/usr/local/RH6_apps/swig-3.0.12 --with-pcre-prefix=/usr/local/RH6_apps/pcre-8.40 --without-clisp --without-maximum-compile-warnings
# make
# make install
by

Configuring 2 Gateways on the Same Linux Box, one for All Traffic, the other for Specific Traffic

To configure two gateways on a Linux machine—one for all traffic (default gateway) and the other for a specific network, you have to consider static routes and multiple network interfaces

Suppose you have 2 networks on a PC:

  1. Default Gateway (GW1) – For General Internet Traffic, 192.168.90.0/24 (eth0).
    • eth0 (Connected to GW1: 192.168.90.1
  2. Specific Gateway (GE2) – For Specific Traffic for AD Authentication, 192.168.0.1/24 (eth1)
    • eth1 (Connected to GW2: 192.168.1.1)

Set the Default Gateway (GW1)

You can set the default gateway for all traffic using the following command:

ip route add default via 192.168.90.1 dev eth0

This sends all non-specific traffic through GW1.

Add Static Route for Specific Network Range via GW2

ip route add 172.21.0.0/16 via 192.168.1.1 dev eth1
ip route add 144.21.0.0/16 via 192.168.1.1 dev eth1

This ensures traffic for 172.21.0.0/16 and 144.21.0.0/16 is routed via GW2.

Verify the Route

ip route show
default via 192.168.90.1 dev eth0 proto static metric 102 
172.21.0.0/16 via 192.168.1.1 dev eth1 proto static metric 103 
144.21.0.0/16 via 192.168.1.1 dev eth1 proto static metric 103
by

Compiling VASP-5.2.12 with Intel MPI-5.0.3

Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

A. Prerequisites

To Compile VASP-5.2.12, I used

  1. Intel Compiler 15.0.6
  2. Intel MPI 5.0.3
  3. Maths Kernel Library 11.2.4

B. Compiling VASP Libraries

Assuming you have unpacked the VASP files. Here is may make file

.SUFFIXES: .inc .f .F
#-----------------------------------------------------------------------
# Makefile for LINUX NAG f90
#-----------------------------------------------------------------------
# fortran compiler
FC=ifort

# C-preprocessor
#CPP     = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C $*.F >$*.f
CPP      = gcc -E -P -C -DLONGCHAR $*.F >$*.f

CFLAGS = -O
FFLAGS = -Os -FI
FREE   = -FR

DOBJ =  preclib.o timing_.o derrf_.o dclock_.o  diolib.o dlexlib.o drdatab.o

#-----------------------------------------------------------------------
# general rules
#-----------------------------------------------------------------------

libdmy.a: $(DOBJ) linpack_double.o
-rm libdmy.a
ar vq libdmy.a $(DOBJ)

linpack_double.o: linpack_double.f
$(FC) $(FFLAGS) $(NOFREE) -c linpack_double.f

# files which do not require autodouble
lapack_double.o: lapack_double.f
$(FC) $(FFLAGS) $(NOFREE) -c lapack_double.f
lapack_single.o: lapack_single.f
$(FC) $(FFLAGS) $(NOFREE) -c lapack_single.f
#lapack_cray.o: lapack_cray.f
#       $(FC) $(FFLAGS) $(NOFREE) -c lapack_cray.f

.c.o:
$(CC) $(CFLAGS) -c $*.c
.F.o:
$(CPP)
$(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f
.F.f:
$(CPP)
.f.o:
$(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f

C. Compiling VASP

1. Copy the Makefile from makefile.linux_ifc_P4 in the vasp software.

# cp makefile.linux_ifc_P4 Makefile

2. Edit the Makefile
FC

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=mpiifort
# fortran linker
FCL=$(FC)

CPP

CPP    = $(CPP_) -DMPI  -DHOST=\"LinuxIFC\" -DIFC \
        -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \
        -DMPI_BLOCK=64000 -Duse_collective -DscaLAPACK

FFLAGS

MKLROOT=/usr/local/RH6_apps/intel_2015/mkl
MKL_PATH=$(MKLROOT)/lib/intel64
FFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw

OFLAG

#Haswell Architecture
OFLAG=-O3 -xCORE-AVX2

#Sandy-Bridge Architecture
OFLAG=-O3

The -xCORE-AVX2 is for Haswell Architecture

BLAS

BLAS= -mkl=cluster

-mkl=cluster is an Intel compiler flag that to include Intel MKL libraries, that will link with Intel MKL BLAS, LAPACK, FFT, ScaLAPACK functions that are used in VASP.

FFT3D

fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
INCS = -I$(MKLROOT)/include/fftw

LAPACK and SCALAPACK

LAPACK=
SCA=

Since the -mkl=cluster, includes MKL ScaLAPACK libraries also, it is not required to mentioned the ScaLAPACK libs. That include LAPACK

References:

  1. Building VASP* with Intel® MKL and Intel® Compilers
  2. (Intel Developer Zone)
by

How to find which switch the OpenSM is running on

1. On any clients do a ibstat

CA type: MT26428
        Number of ports: 2
        Firmware version: 2.9.1000
        Hardware version: b0
        Node GUID: 0x000xxxxxxxxxx
        System image GUID: 0xxxxxxxxxxxxxxxx
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 184
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510868
                Port GUID: 0x0002c903005abfd7
                Link layer: InfiniBand

2. To check for the SM Manager

# smpquery ND -L 1
Node Description:.......Voltaire 4036 # 4036-0D9E

where 1 is the SM Lid:1

3. If you query InfiniBand SMInfo attribute

# sminfo
sminfo: sm lid 1 sm guid 0x8f10500200d9e, activity count 42554 priority 4 state 3 SMINFO_MASTER

 

by

Removing Existing NSD Nodes from GPFS Cluster

Removing existing NSD Nodes from the GPFS Cluster is not difficult, but there are several steps to take note of.

Step 1: Check and make sure the NSD Nodes you are removing are not quorum-manager.

See Removing Quorum Manager from NSD Nodes

Step 2: Check that Quorum-Manager have been removed from the old NSD

# mmlscluster

GPFS cluster information
========================
GPFS cluster name:         mygpfs.gpfsnsd1
GPFS cluster id:           720691660936079521
GPFS UID domain:           mygpfs.gpfsnsd1
Remote shell command:      /usr/bin/ssh
Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
Primary server:    newnsd3
Secondary server:  newnsd4
.....
.....
.....
.....
714   oldnsd1          192.168.111.5   oldnsd1
715   oldnsd2          192.168.111.6   oldnsd2
716   newnsd3          192.168.111.7   newnsd3         quorum-manager
717   newnsd4          192.168.111.8   newnsd4         quorum-manager

Step 3a: Umount the GPFS File System

# mmumount all -a

Step 3b: Check all the GPFS nodes have been unmounted

# mmlsmount all

File system gpfs0 is mounted on 0 nodes.

Step 4: Displays Network Shared Disk (NSD) information for the GPFS cluster.

# mmlsnsd

File system   Disk name    NSD servers
---------------------------------------------------------------------------
gpfs0         dcs3700A_2   File system   Disk name    NSD servers
---------------------------------------------------------------------------
gpfs0         dcs3700A_2   newnsd3,oldnsd1
gpfs0         dcs3700A_3   newnsd3,oldnsd1
gpfs0         dcs3700A_4   newnsd3,oldnsd1
gpfs0         dcs3700A_5   newnsd3,oldnsd1
gpfs0         dcs3700A_6   newnsd3,oldnsd1
gpfs0         dcs3700A_7   newnsd3,oldnsd1
gpfs0         dcs3700B_2   newnsd4,oldnsd2
gpfs0         dcs3700B_3   newnsd4,oldnsd2
gpfs0         dcs3700B_4   newnsd4,oldnsd2
gpfs0         dcs3700B_5   newnsd4,oldnsd2
gpfs0         dcs3700B_6   newnsd4,oldnsd2
gpfs0         dcs3700B_7   newnsd4,oldnsd2

Step 5: Changes Network Shared Disk (NSD) configuration attributes.

# mmchnsd dcs3700A_2:newnsd3

To confirm the changes, issue this command:

# mmlsnsd -d dcs3700A_2
File system   Disk name    NSD servers
---------------------------------------------------------------------------
 gpfs0         dcs3700A_2   newnsd3

Do for the rest…..

# mmchnsd dcs3700A_3:newnsd3
# mmlsnsd -d dcs3700A_3
# mmchnsd dcs3700A_4:newnsd3
# mmlsnsd -d dcs3700A_4
# mmchnsd dcs3700A_5:newnsd3
# mmlsnsd -d dcs3700A_5
# mmchnsd dcs3700A_6:newnsd3
# mmlsnsd -d dcs3700A_6
# mmchnsd dcs3700A_7:newnsd3
# mmlsnsd -d dcs3700A_7
# mmchnsd dcs3700B_2:newnsd4
# mmlsnsd -d dcs3700B_2
# mmchnsd dcs3700B_3:newnsd4
# mmlsnsd -d dcs3700B_3
# mmchnsd dcs3700B_4:newnsd4
# mmlsnsd -d dcs3700B_4
# mmchnsd dcs3700B_5:newnsd4
# mmlsnsd -d dcs3700B_5
# mmchnsd dcs3700B_6:newnsd4
# mmlsnsd -d dcs3700B_6
# mmchnsd dcs3700B_7:newnsd4
# mmlsnsd -d dcs3700B_7

Step 6: Remove Old NSD Nodes

Once you have confirmed that the NSD nodes are ok.

# mmdelnode -N oldnsd1,oldnsd2

Verify that the old NSD nodes are no more in the cluster

# mmlscluster

Step 7: Remount the File System

# mmmount all -a

References:

  1. General Parallel File System
by

LAMMPS Tools and Packmol with Intel Fortran

PACKMOL information can be obtained from http://www.ime.unicamp.br/~martinez/packmol/userguide.shtml#conv

Installing can be found at http://www.ime.unicamp.br/~martinez/packmol/userguide.shtml#comp

1. Compile Packmol with Intel Fortran

# tar -zxvf packmol.tar.gz
# cd packmol
# ./configure ifort
# make

2. LAMMPS Tools

# git clone https://github.com/jdevemy/lammps-tools.git
# cd lammps-tools
# python setup.py build
# sudo python setup.py install

3. Make sure the Python has the following libraries in create_conf (sys, os, logging, argparse, math random)

4. Make sure the Python (if you install lammps-tool)

# export PYTHONPATH=/home/user1/Downloads/lammps-tools-master/lib
# ./create_conf