Month: October 2016

by

How to find which switch the OpenSM is running on

1. On any clients do a ibstat

CA type: MT26428
        Number of ports: 2
        Firmware version: 2.9.1000
        Hardware version: b0
        Node GUID: 0x000xxxxxxxxxx
        System image GUID: 0xxxxxxxxxxxxxxxx
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 184
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510868
                Port GUID: 0x0002c903005abfd7
                Link layer: InfiniBand

2. To check for the SM Manager

# smpquery ND -L 1
Node Description:.......Voltaire 4036 # 4036-0D9E

where 1 is the SM Lid:1

3. If you query InfiniBand SMInfo attribute

# sminfo
sminfo: sm lid 1 sm guid 0x8f10500200d9e, activity count 42554 priority 4 state 3 SMINFO_MASTER

 

Advertisement
by

Removing Existing NSD Nodes from GPFS Cluster

Removing existing NSD Nodes from the GPFS Cluster is not difficult, but there are several steps to take note of.

Step 1: Check and make sure the NSD Nodes you are removing are not quorum-manager.

See Removing Quorum Manager from NSD Nodes

Step 2: Check that Quorum-Manager have been removed from the old NSD

# mmlscluster

GPFS cluster information
========================
GPFS cluster name:         mygpfs.gpfsnsd1
GPFS cluster id:           720691660936079521
GPFS UID domain:           mygpfs.gpfsnsd1
Remote shell command:      /usr/bin/ssh
Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
Primary server:    newnsd3
Secondary server:  newnsd4
.....
.....
.....
.....
714   oldnsd1          192.168.111.5   oldnsd1
715   oldnsd2          192.168.111.6   oldnsd2
716   newnsd3          192.168.111.7   newnsd3         quorum-manager
717   newnsd4          192.168.111.8   newnsd4         quorum-manager

Step 3a: Umount the GPFS File System

# mmumount all -a

Step 3b: Check all the GPFS nodes have been unmounted

# mmlsmount all

File system gpfs0 is mounted on 0 nodes.

Step 4: Displays Network Shared Disk (NSD) information for the GPFS cluster.

# mmlsnsd

File system   Disk name    NSD servers
---------------------------------------------------------------------------
gpfs0         dcs3700A_2   File system   Disk name    NSD servers
---------------------------------------------------------------------------
gpfs0         dcs3700A_2   newnsd3,oldnsd1
gpfs0         dcs3700A_3   newnsd3,oldnsd1
gpfs0         dcs3700A_4   newnsd3,oldnsd1
gpfs0         dcs3700A_5   newnsd3,oldnsd1
gpfs0         dcs3700A_6   newnsd3,oldnsd1
gpfs0         dcs3700A_7   newnsd3,oldnsd1
gpfs0         dcs3700B_2   newnsd4,oldnsd2
gpfs0         dcs3700B_3   newnsd4,oldnsd2
gpfs0         dcs3700B_4   newnsd4,oldnsd2
gpfs0         dcs3700B_5   newnsd4,oldnsd2
gpfs0         dcs3700B_6   newnsd4,oldnsd2
gpfs0         dcs3700B_7   newnsd4,oldnsd2

Step 5: Changes Network Shared Disk (NSD) configuration attributes.

# mmchnsd dcs3700A_2:newnsd3

To confirm the changes, issue this command:

# mmlsnsd -d dcs3700A_2
File system   Disk name    NSD servers
---------------------------------------------------------------------------
 gpfs0         dcs3700A_2   newnsd3

Do for the rest…..

# mmchnsd dcs3700A_3:newnsd3
# mmlsnsd -d dcs3700A_3
# mmchnsd dcs3700A_4:newnsd3
# mmlsnsd -d dcs3700A_4
# mmchnsd dcs3700A_5:newnsd3
# mmlsnsd -d dcs3700A_5
# mmchnsd dcs3700A_6:newnsd3
# mmlsnsd -d dcs3700A_6
# mmchnsd dcs3700A_7:newnsd3
# mmlsnsd -d dcs3700A_7
# mmchnsd dcs3700B_2:newnsd4
# mmlsnsd -d dcs3700B_2
# mmchnsd dcs3700B_3:newnsd4
# mmlsnsd -d dcs3700B_3
# mmchnsd dcs3700B_4:newnsd4
# mmlsnsd -d dcs3700B_4
# mmchnsd dcs3700B_5:newnsd4
# mmlsnsd -d dcs3700B_5
# mmchnsd dcs3700B_6:newnsd4
# mmlsnsd -d dcs3700B_6
# mmchnsd dcs3700B_7:newnsd4
# mmlsnsd -d dcs3700B_7

Step 6: Remove Old NSD Nodes

Once you have confirmed that the NSD nodes are ok.

# mmdelnode -N oldnsd1,oldnsd2

Verify that the old NSD nodes are no more in the cluster

# mmlscluster

Step 7: Remount the File System

# mmmount all -a

References:

  1. General Parallel File System
by

LAMMPS Tools and Packmol with Intel Fortran

PACKMOL information can be obtained from http://www.ime.unicamp.br/~martinez/packmol/userguide.shtml#conv

Installing can be found at http://www.ime.unicamp.br/~martinez/packmol/userguide.shtml#comp

1. Compile Packmol with Intel Fortran

# tar -zxvf packmol.tar.gz
# cd packmol
# ./configure ifort
# make

2. LAMMPS Tools

# git clone https://github.com/jdevemy/lammps-tools.git
# cd lammps-tools
# python setup.py build
# sudo python setup.py install

3. Make sure the Python has the following libraries in create_conf (sys, os, logging, argparse, math random)

4. Make sure the Python (if you install lammps-tool)

# export PYTHONPATH=/home/user1/Downloads/lammps-tools-master/lib
# ./create_conf
by

Compiling MEEP with Intel-15.0.6, Intel-MPI 5.0.3 and HDFT-1.8.17

Meep (or MEEP) is a free finite-difference time-domain (FDTD) simulation software package developed at MIT to model electromagnetic systems, along with our MPB eigenmode package. The latest official version is 1.3 and can be found at  Download Page for Meep

Before you compile Meep 1.2.1, you need to first compile the libctl library. Compiling the libctl library is quite straightforward. After downloading,

Step 1: Compiling libctl-3.2.1

# tar -zxvf libctl-3.2.1.tar.gz
# cd libctl-3.2.1
# ./configure --prefix=/usr/local/libctl-3.2.1
# make -j8
# make install

Step 2: Other Prerequisites include guile and guile-devel. Do make sure you install these 2 packages which can be done

# yum install guile guile-devel

Step 3: Prepare Intel Compilers and Intel MPI environment

$ vim .bashrc
source /usr/local/intel_2015/bin/compilervars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64
export CC=icc
export CXX=icpc
export F77=ifort
export MPICC=mpicc
export MPICXX=mpiicpc
export CFLAGS="-O3 -xHost -fno-alias -align"
export FFLAGS="-O3 -xHost -fno-alias -align"
export CXXFLAGS="-O3 -xHost -fno-alias -align"
export FFlags="-I/usr/local/intel_2015/impi/5.0.3.049/include64 -L/usr/local/intel_2015/impi/5.0.3.049/lib64"

Step 4: Compiling hdf5-1.8.17

See Compiling HDF5-1.8.17 with Intel-15.0.6 and Intel-MPI-5.0.6

Step 5: Compiling MEEP-1.3

$ ./configure --prefix=/usr/local/meep-1.3.1_impi-5.0.3 --with-mpi \
--with-libctl="/usr/local/libctl-3.2.1/share/libctl" \
LDFLAGS="-L/usr/local/libctl-3.2.1/lib -L/usr/local/hdf5-1.8.17/lib" \
CPPFLAGS="-I/usr/local/libctl-3.2.1/include -I/usr/local/hdf5-1.8.17/include"
$ make -j 12
$ make install