Recommended Sites for Linux Command Lines
Compiling VASP 5.3.3 with OpenMPI 1.6.5 and Intel 12.1.5
Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.
To compile VASP, I will be using the following
- Intel Compiler XE 12.1.5
- OpenMPI 1.6.5
- Maths Kernel Library (MKL)
- CentOS 6.3 / RH 6.3
A. Prerequisites
Step 1: Install and Compile Intel XE Compilers 12.1.5 and OpenMPI 1.6.5
Step 2: Intel MKL FFTW wrappers
B. Compiling VASP Libraries
Assuming you have unpacked the VASP files. Here is may make file
.SUFFIXES: .inc .f .F #----------------------------------------------------------------------- # Makefile for LINUX NAG f90 #----------------------------------------------------------------------- # fortran compiler FC=ifort # C-preprocessor #CPP = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C $*.F >$*.f CPP = gcc -E -P -C -DLONGCHAR $*.F >$*.f CFLAGS = -O FFLAGS = -Os -FI FREE = -FR DOBJ = preclib.o timing_.o derrf_.o dclock_.o diolib.o dlexlib.o drdatab.o #----------------------------------------------------------------------- # general rules #----------------------------------------------------------------------- libdmy.a: $(DOBJ) linpack_double.o -rm libdmy.a ar vq libdmy.a $(DOBJ) linpack_double.o: linpack_double.f $(FC) $(FFLAGS) $(NOFREE) -c linpack_double.f # files which do not require autodouble lapack_double.o: lapack_double.f $(FC) $(FFLAGS) $(NOFREE) -c lapack_double.f lapack_single.o: lapack_single.f $(FC) $(FFLAGS) $(NOFREE) -c lapack_single.f #lapack_cray.o: lapack_cray.f # $(FC) $(FFLAGS) $(NOFREE) -c lapack_cray.f .c.o: $(CC) $(CFLAGS) -c $*.c .F.o: $(CPP) $(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f .F.f: $(CPP) .f.o: $(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f
C. Compiling VASP
1. Copy the Makefile from makefile.linux_ifc_P4 in the vasp software.
# cp makefile.linux_ifc_P4 Makefile
2. Edit the Makefile
FC (~ line 62)
#----------------------------------------------------------------------- # fortran compiler and linker #----------------------------------------------------------------------- FC=mpif90 # fortran linker FCL=$(FC) -mkl
FFLAGS (~line 108)
#----------------------------------------------------------------------- # general fortran flags (there must a trailing blank on this line) # byterecl is strictly required for ifc, since otherwise # the WAVECAR file becomes huge #----------------------------------------------------------------------- FFLAGS = -FR -names lowercase -assume byterecl
BLAS (~ line 149)
# BLAS # setting -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines usually speeds up program execution # BLAS= -Wl,--start-group $(MKL_PATH)/libmkl_intel_lp64.a $(MKL_PATH)/libmkl_intel_thread.a $(MKL_PATH)/libmkl_core.a -Wl,--end-group -lguide # faster linking and available from at least version 11 BLAS= -mkl=sequential
LAPACK (~ line 158)
# LAPACK from mkl, usually faster and contains scaLAPACK as well #LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a LAPACK =
CPP ( ~ line 223)
#----------------------------------------------------------------------- CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \ -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \ -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK \ -DRPROMU_DGEMV -DRACCMU_DGEMV
SCALAPACK (~ line 233)
#----------------------------------------------------------------------- # location of SCALAPACK # if you do not use SCALAPACK simply leave this section commented out #----------------------------------------------------------------------- # usually simplest link in mkl scaLAPACK BLACS= -lmkl_blacs_openmpi_lp64 SCA= $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
LIBRARIES (~line 238)
#----------------------------------------------------------------------- # libraries #----------------------------------------------------------------------- LIB = -L../vasp.5.lib -ldmy \ ../vasp.5.lib/linpack_double.o \ $(SCA) $(LAPACK) $(BLAS)
Parallel FFT (~ line 246)
#----------------------------------------------------------------------- # parallel FFT #----------------------------------------------------------------------- # FFT: fftmpi.o with fft3dlib of Juergen Furthmueller FFT3D = fftmpi.o fftmpi_map.o fft3dfurth.o fft3dlib.o
References:
Compiling GCC 4.7.2 on CentOS 5
Step 1: Download the following prerequistics applications libraries from ftp://gcc.gnu.org/pub/gcc/infrastructure/
- gmp-4.3.2.tar.bz2
- mpfr-2.4.2.tar.bz2
- mpc-0.8.1.tar.gz
1. Install gmp-4.3.2
# bunzip2 gmp-4.3.2.tar.bz2 # tar -zxvf gmp-4.3.2.tar # cd gmp-4.3.2 # ./configure --prefix=/usr/local/gmp-4.3.2 # make # make install
2. Install mpfr-2.4.2 (requires gmp-4.3.2 as prerequisites)
# bunzip2 mpfr-2.4.2.tar.bz2 # tar -zxvf mpfr-2.4.2.tar # cd mpfr-2.4.2 # ./configure --prefix=/usr/local/mpfr-2.4.2 --with-gmp=/usr/local/gmp-4.3.2/ # make # make install
3. Install mpc-0.8.1 (requires gmp-4.3.2 and mpfr-2.4.2 as prerequisites )
# tar -zxvf mpc-0.8.1.tar.gz # cd mpc-0.8.1 #./configure --prefix=/usr/local/mpc-0.8.1/ --with-gmp=/usr/local/gmp-4.3.2/ --with-mpfr=/usr/local/mpfr-2.4.2 # make # make install
4. Update your LD_LIBRARY_PATH at your ~/.bashrc
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/mpc-0.8.1:/usr/local/mpfr-2.4.2/lib:/usr/local/gmp-4.3.2/lib
5. Install the glibc-devel.i686. For more information, do look at Error when compiling GCC 4.8.1 (linuxtoolkit.blogspot.com)
6. Finally install GNU CC 4.7.2
# tar -zxvf gcc-4.7.2.tar.gz # cd gcc-4.7.2 # mkdir build-gcc # cd build-gcc # ../configure --prefix=/usr/local/gcc-4.7.2 \ --with-mpfr=/usr/local/mpfr-2.4.2 \ --with-mpc=/usr/local/mpc-0.8.1 \ --with-gmp=/usr/local/gmp-4.3.2 \ --with-mpfr-include=/usr/local/mpfr-2.4.2/include \ --with-mpc-include=/usr/local/mpc-0.8.1/include # make all-gcc # make install-gcc
Compiling OpenMPI 1.6.5 with Intel 12.1.5 on CentOS 6
Step 1: Download the OpenMPI Software from http://www.open-mpi.org/ . The current stable version at point of writing is OpenMPI 1.6.5
Step 2: Download and Install the Intel Compilers from Intel Website. More information can be taken from Free Non-Commercial Intel Compiler Download
Step 3: Add the Intel Directory Binary Path to the Bash Startup
At my ~/.bash_profile directory, I've added source /usr/local/intel/composerxe/bin/compilervars.sh intel64 export PATH=$PATH:/usr/local/intel/composerxe/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/RH6_apps/intel/composerxe/lib/intel64:/usr/local/intel/composerxe/mkl/lib/intel64
At command prompt
# source .bashrc
Step 4: Configuration Information
# gunzip -c openmpi-1.6.5.tar.gz tar xf - # cd openmpi-1.6.5 # /configure --prefix=/usr/local/openmpi-1.6.5-intel-v12.1.5 CC=icc CXX=icpc F77=ifort FC=ifort --with-devel-headers --enable-binaries # make all install
You will need to include –devel-headers –enable-binaries” or you will face the error Cannot open OpenMPI Configuration mpif90-wrapper-data.txt
Step 5: Test the configuration
$ mpicc -v cc version 12.1.5 (gcc version 4.4.6 compatibility)
Compile and Build FFTW3 wrapper routines on MKL implementation of FFT
Software like VASP uses calling statements of the FFTW3 which differ to the implementation of FFT in MKL. You will need to compile the wrapper routines for MKL
# cd $MKLROOT/interfaces/fftw3xf
# makeAfter a successful compilation libfftw3xf_intel.a was created in this directory
Intel Enterprise Edition for Lustre Software (White Papers)
Building OpenMPI Libraries for 64-bit integers
There is an excellent article on how to build OpenMPI Libraries for 64-bit integers. For more detailed information, do look at How to build MPI libraries for 64-bit integers
The information on this website is taken from the above site.
Step 1: Check the integer size. Do the following. If the output is as below, you have to compile OpenMPI with 64 bits.
# ompi_info -a | grep 'Fort integer size' Fort integer size: 4
Intel Compilers Step 2a: To compile OpenMPI with Intel Compilers and with 64-bits integers, do the following:
# ./configure --prefix=/usr/local/openmpi CXX=icpc CC=icc F77=ifort FC=ifort FFLAGS=-i8 FCFLAGS=-i8 # make -j 8 # make install
* GNU Compilers
Step 2b: To compile OpenMPI with GNU Compilers and with 64-bits, do the followings:
# ./configure --prefix=/usr/local/openmpi CXX=g++ CC=gcc F77=gfortran FC=gfortran \ FFLAGS="-m64 -fdefault-integer-8" \ CFLAGS="-m64 -fdefault-integer-8" \ CFLAGS=-m64 \ CXXFLAGS=-m64 # make -j 8 # make install
Step 3: Update your PATH and LD_LIBRARY_PATH in your .bashrc
export $PATH=/usr/local/openmpi/bin:$PATH export $LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH
Verify that the installation is correct
# ompi_info -a | grep 'Fort integer size' Fort integer size: 8
Registering sufficent memory for OpenIB when using Mellanox HCA
If you encountered errors like “error registering openib memory” similar to what is written below. You may want to take a look at the OpenMPI FAQ – I’m getting errors about “error registering openib memory”; what do I do? .
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory. This can cause MPI jobs to
run with erratic performance, hang, and/or crash.
This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered. You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.
See this Open MPI FAQ item for more information on these Linux kernel module
parameters:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Local host: node02
Registerable memory: 32768 MiB
Total memory: 65476 MiB
Your MPI job will continue, but may be behave poorly and/or hang.
The explanation solution can be found at How to increase MTT Size in Mellanox HCA.
In summary, the error occurred when applications which consumed a large amount of memory, application might fail when not enough memory can be registered with RDMA. There is a need to increase MTT size. But increasing MTT size hasve the downside of increasing the number of “cache misses” and increases latency.
1. To check your value of log_num_mtt
# cat /sys/module/mlx4_core/parameters/log_num_mtt
2. To check your value of log_mtts_per_seg
# cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
There are 2 parameters that affect registered memory. This can be taken from http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
With Mellanox hardware, two parameters are provided to control the size of this table:
- log_num_mtt (on some older Mellanox hardware, the parameter may be
num_mtt, notlog_num_mtt): number of memory translation tables - log_mtts_per_seg:
The amount of memory that can be registered is calculated using this formula:
In newer hardware:
max_reg_mem = (2^log_num_mtt) * (2^log_mtts_per_seg) * PAGE_SIZE
In older hardware:
max_reg_mem = num_mtt * (2^log_mtts_per_seg) * PAGE_SIZE
For example if your server’s Physical Memory is 64GB RAM. You will need to registered 2 times the Memory (2x64GB=128GB) for the max_reg_mem. You will also need to get the PAGE_SIZE (See Virtual Memory PAGESIZE on CentOS)
max_reg_mem = (2^ log_num_mtt) * (2^3) * (4 kB) 128GB = (2^ log_num_mtt) * (2^3) * (4 kB) 2^37 = (2^ log_num_mtt) * (2^3) * (2^12) 2^22 = (2^ log_num_mtt) 22 = log_num_mtt
The setting is found in /etc/modprobe.d/mlx4_mtt.conf for every nodes.
References:
Tracking Batch Jobs at Platform LSF
The content article is taken from http://users.cs.fiu.edu/~tho01/psg/3rdParty/lsf4_userGuide/07-tracking.html
1. Displaying All Job Status
# bjobs -u all
2. Report Reasons why a job is pending
# bjobs -p
3. Report Pending Reasons with host names for each conditions
# bjobs -lp
4. Detailed Report on a specific jobs
# bjobs -l 6653
5. Reasons why the job is suspended
# bjobs -s
6. Displaying Job History
# bpeek 12345
7. Killing Jobs
# bkill 12345
8. Stop the Job
# bstop 12345
9 Resume the job
# bresume 12345
Installing pdsh to issue commands to a group of nodes in parallel in CentOS
1. What is pdsh?
Pdsh is a high-performance, parallel remote shell utility. It uses a sliding window of threads to execute remote commands, conserving socket resources while allowing some connections to timeout if needed. It was originally written as a replacement for IBM’s DSH on clusters at LLNL. More information can be found at PDSH Web site
2. Setup EPEL yum repository on CentOS 6.
For more information, see Repository of CentOS 6 and Scientific Linux 6
3. Do a yum install
# yum install pdsh
To confirm installation
# which pdsh
4. Configure user environment for PDSH
# vim /etc/profile.d/pdsh.sh
Edit the following:
# setup pdsh for cluster users export PDSH_RCMD_TYPE='ssh' export WCOLL='/etc/pdsh/machines'
5. Put the host name of the Compute Nodes
# vim /etc/pdsh/machines/
node1 node2 node3 ....... .......
6. Make sure the nodes have their SSH-Key Exchange. For more information, see Auto SSH Login without Password
7. Do Install Step 1 to Step 3 on ALL the client nodes.
B. USING PDSH
Run the command ( pdsh [options]… command )
1. To target all the nodes found at /etc/pdsh/machinefile. Assuming the files are transferred already. Do note that the parallel copy comes with the pdsh utilities
# pdsh -a "rpm -Uvh /root/htop-1.0.2-1.el6.rf.x86_64.rpm"
2. To target specific nodes, you may want to consider using the -x command
# pdsh -x host1,host2 "rpm -Uvh /root/htop-1.0.2-1.el6.rf.x86_64.rpm"
References
The Linux Cluster 