Month: May 2010

by

Basic xCAT installation

This Blog entry is modified from Basic Install xCAT from xCAT wiki with some minor modification at point 7*, 8* and 9*

1. Pre-install

  • Internal IP of headnode – referred to as (xcat_int_ip)
  • External IP (internet connected) of headnode – referred to as (xcat_ext_ip)
  • External DNS server IP – referred to as (dns_ext_ip)
  • Cluster domain – referred to as (cluster_dom)

2. Network Wiring

  • eth0 is attached into an existing corporate network
  • eth1 is attached to the switch that the compute nodes are attached to

3. Setup Networking
Configure the Ethernet interfaces

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
PEERDNS=no
BOOTPROTO=dhcp
HWADDR=00:14:5E:6B:18:21
ONBOOT=yes

ifdown eth0 && ifup eth0

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=static
HWADDR=00:14:5E:6B:18:22
ONBOOT=yes
IPADDR=(xcat_int_ip)
NETMASK=255.255.255.0

ifdown eth1 && ifup eth1

4. Install xCAT
Add the xCAT package repositories

cd /etc/yum.repos.d
wget http://xcat.sourceforge.net/yum/xcat-core/xCAT-core.repo
wget http://xcat.sourceforge.net/yum/xcat-dep/rh5/x86_64/xCAT-dep.repo
yum clean metadata
yum install xCAT

Verify the install

source /etc/profile.d/xcat.sh
tabdump site

If the tabdump command works, xCAT is installed. If it doesn’t work, check to ensure all previous steps completed sucessfully

5. Configure the xCAT site table.

  1. Set the dns to forward requests for the (dns_ext_ip) network
  2. Set the domain to (cluster_dom),
  3. Set the master and nameserver to (dns_ext_ip),
  4. Set eth1 to be the dhcp server interface
tabedit site

#key,value,comments,disable
"xcatdport","3001",,
"xcatiport","3002",,
"tftpdir","/tftpboot",,
"master","(xcat_int_ip)",,
"domain","(cluster_dom)",,
"installdir","/install",,
"timezone","America/Denver",,
"nameservers","(xcat_int_ip)",,
"forwarders","(dns_ext_ip)"
"dhcpinterfaces","eth1"
"ntpservers","0.north-america.pool.ntp.org"

6. Setup the xCAT networks table

tabedit networks

#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,dynamicrange,nodehostname,comments,disable
internal,"10.10.10.0","255.255.255.0","eth1","10.10.10.1","10.10.10.1","10.10.10.1","10.10.10.1",,,"10.10.10.200-10.10.10.254",,,
external,"192.168.0.0","255.255.0.0","eth0",,,,"192.168.0.1",,,,

There should be an entry for each network the nodes need to access. In this case, DNS is forwarded to the 192.168.0.1 server:

* 7. Setup the xCAT noderes table
(variation from xCAT wiki)
The noderes table defines the resources of the nodes. This includes all of the servers it uses to boot to a usable state and all the types of boot ups it will do.
Note: Primary interface (primarynic) for ipmi machines is the one where the BMC is set to. 

tabedit noderes

#node,servicenode,netboot,tftpserver,nfsserver,monserver,nfsdir,installnic,primarynic,discoverynics,cmdinterface,xcatmaster,current_osimage,next_osimage,nimserver,comments,disable
"compute",,"pxe","10.10.10.1","10.10.10.1",,"/install","eth0","eth0",,,,,,,,

*8. Setup the xCAT passwd table
(variation from xCAT wiki)

tabedit passwd

#key,username,password,comments,disable
"omapi","xcat_key","xxxxxxxxxx=",,
"system","root","passw0rd",,

The “omapi” is generated by xCAT. Don’t touch it. But you will need to include the 2nd line “system”

*9. Setup the xCAT chain table
(variation from xCAT wiki)

tabedit chain

#node,currstate,currchain,chain,ondiscover,comments,disable
"n00",,,,,,
"n01",,,,,,
"n02",,,,,,

*10. Setup the xCAT nodetype table
(variation from xCAT wiki)

tabedit nodetype

#node,os,arch,profile,provmethod,supportedarchs,nodetype,comments,disable
"n00","centos5.4","x86_64","compute",,,,,
"n01","centos5.4","x86_64","compute",,,,,
"n02","centos5.4","x86_64","compute",,,,,

11. Setup the xCAT hosts table

tabedit hosts

xcat,(xcat_int_ip)
"n00",(n00_ip)
"n01",(n01_ip)
"n02",(n02_ip)

12. Setup the xCAT mac table

tabedit mac

"n00","eth0",(mac)
"n01","eth0",(mac)
"n02","eth0",(mac)

13. Setup the xCAT nodelist table

tabedit nodelist

"n00","compute,all",,,
"n01","compute,all",,,
"n02","compute,all",,,

14. Setup the xCAT nodehm table

tabedit nodehm

"n00","ipmi","ipmi",,,,,,,,,,
"n01","ipmi","ipmi",,,,,,,,,,
"n02","ipmi","ipmi",,,,,,,,,,

15. Create the hosts file

makehosts all

There should entries in the /etc/hosts that reflect all the nodes

16. Create the DHCP files

makedhcp -n
makedhcp all
service dhcpd restart
chkconfig --level 345 dhcpd on

assuming all the nodes and devices are in the “all” group, that command will work:
Note: dhcpd does NOT need to be restarted after adding a node via makedhcp, but does after running the “-n” option which creates a new file

17. Edit /etc/resolv.conf

vi /etc/resolv.conf
search (xcat_dom)
nameserver (xcat_int_ip)

18. Build the DNS server

makedns
makedns all
service named restart
chkconfig --level 345 named on

assuming all the nodes and devices are in the “all” group, this command will work:
Note: named DOES need to be restarted after running a makedns command

19. Routing to the Internet through the Head Node

echo "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE" >> /etc/rc.local
echo "echo 1 > /proc/sys/net/ipv4/ip_forward" >> /etc/rc.local

* Do remember to configure the gateway for each of the compute node to eth1 (private nic) of the Head Node. Go to client private nic /etc/sysconfig/network-scripts/ifcfg-eth0 and add “GATEWAY=(int_ip_address)”

The xCAT server should now be completely configured.

20. Setup Images

copycds CentOS-5.2-i386-bin-DVD.iso

Note: Do this for the DVD ISO and ”’NOT”’ the cd!

21. Install the node!

rinstall n01

That’s not all. See Using xCAT contributed scripts

by

Building OpenMPI with Intel Compilers

Modified from Performance Tools for Software Developers – Building Open MPI* with the Intel® compilers

Step 1: Download the OpenMPI Software from http://www.open-mpi.org/ . The current stable version at point of writing is OpenMPI 1.3.2

Step 2: Download and Install the Intel Compilers from Intel Website. More information can be taken from Free Non-Commercial Intel Compiler Download

Step 3: Add the Intel Directory Binary Path to the Bash Startup

At my ~/.bash_profile directory, I’ve added

export PATH=$PATH:/opt/intel/Compiler/11.0/081/bin/intel64

At command prompt

# source .bashrc

Step 4: Configuration Information

# source /opt/intel/Compiler/11.0/081/bin/compilervars.sh
# gunzip -c openmpi-1.2.tar.gz tar xf -
# cd openmpi-1.2
#./configure --prefix=/usr/local CC=icc CXX=icpc F77=ifort FC=ifort
# make all install

Step 5: Setting PATH environment for OpenMPI
At my ~/.bash_profile directory, I’ve added.

export PATH=/usr/local/bin:${PATH} 
export LD_LIBRARY_PATH=/opt/intel/Compiler/11.0/081/lib/intel64:${LD_LIBRARY_PATH}
(The LD_LIBRARY_PATH must point to /opt/intel/Compiler/11.0/081/lib/intel64/libimf.so)

Step 6: test

$ mpicc --v
cc version 12.1.5 (gcc version 4.4.6 compatibility)
by

Installing NWChem 5 with OpenMPI, Intel Compilers and MKL and CentOS 5.x

With much credit to Vallard Land’s Blog on Compiling NWChem and information on NwChem Build Notes from CSE Wiki. I was able to install NwChem on my GE-interconnect cluster with minimal modification. First install the prerequistics, that is Intel Compilers and MKL and of course OpenMPI. I’m using CentOS 5.4 x86-64

  1. If you are eligible for the Intel Compiler Free Download. Download the Free Non-Commercial Intel Compiler Download
  2. Build OpenMPI with Intel Compiler

Finally, the most important, the installation of NWChem. First go to NWChem, read the terms and conditions and request for a login and password. Once you have obtained the tar copy of NwChem. At this point in time, download “nwchem-5.1.1.tar.tar”

# tar -xvf nwchem-5.1.1.tar.tar
# cd nwchem-5.1.1

Create a script so that all these “export” parameter can be typed once only and kept. The script I called it compile_nwchem.sh. Make sure that the ssh key are exchanged between the nodes. To have idea an of SSH key exchange, see blog entry Auto SSH Login without Password

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/melvin/nwchem-5.1.1/   (installation path)
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/usr/local/
export MPI_LIB=$MPI_LOC/lib
export LIBMPI="-L $MPI_LIB -lmpi -lopen-pal -lopen-rte -lmpi_f90 -lmpi_f77"
export MPI_INCLUDE=$MPI_LOC/include
# export ARMCI_NETWORK=OPENIB (if you using IB)
export LARGE_FILES=TRUE
export NWCHEM_MODULES=all
export FC=ifort
export CC=icc

cd $NWCHEM_TOP/src
make CC=icc FC=ifort -j8

it should compiled well without issue. You should have nwchem executable Do note that NWCHEM is the final binary path of usage. NWCHEM_TOP is

# export NWCHEM=/usr/local/nwchem-5.1.1
# export NWCHEM_TOP=/home/melvin/nwchem-5.1.1/

# mkdir $NWCHEM/bin $NWCHEM/data
# cp /home/melvin/nwchem-5.1.1/bin/LINUX64/nwchem $NWCHEM/bin
# cp /home/melvin/nwchem-5.1.1/bin/LINUX64/depend.x $NWCHEM/bin/
# cd $NWCHEM_TOP/src/basis
# cp -r libraries $NWCHEM/data/
# cd $NWCHEM_TOP/src/
# cp -r data $NWCHEM
# cd $NWCHEM_TOP/src/nwpw/libraryps
# cp -r pspw_default $NWCHEM/data/
# cp -r paw_default/ $NWCHEM/data/
# cp -r TM $NWCHEM/data/
# cp -r HGH_LDA $NWCHEM/data/

This should put complete. Make sure the $NWCHEM directory is made available to the rest of the cluster

Finally, copy the src to the

# cp -r /home/melvin/nwchem-5.1.1/bin/LINUX64/nwchem/src $NWCHEM/src

Another good resource can be seen How to build Nwchem-5.1.1 on Intel Westmere with Infiniband network

by

Building the GAMESS with Intel® Compilers, Intel® MKL and OpenMPI on Linux

Modified from the excellent tutorial Building the GAMESS with Intel® Compilers, Intel® MKL and Intel® MPI on Linux for OpenMPI which is

The prerequisites Software

  1. Intel® C++ Compiler for LINUX,
  2. Intel® Fortran Compiler for LINUX,
  3. Intel® MKL,
  4. OpenMPI for Linux.

Platform:

  1. IA64/x86_64.

Installing the Prerequisites

  1. If you are eligible for the Intel Compiler Free Download. Download the Free Non-Commercial Intel Compiler Download
  2. Compile Intel Compilers with OpenMPI. See Building OpenMPI with Intel Compiler. Make sure your pathing are properly written and sourced.

Intel Environment setup
I created a intel.sh script inside /etc/profile.d/ and put the following information inside

# cd /etc/profile.d
# touch intel.sh
# vim intel.sh

Edit the following

export INTEL_COMPILER_TOPDIR="/opt/intel/Compiler/11.1/069"
. $INTEL_COMPILER_TOPDIR/bin/intel64/ifortvars_intel64.sh
. $INTEL_COMPILER_TOPDIR/bin/intel64/iccvars_intel64.sh

Building the Application

1. Copy/move tar file gamess-current.tar.gz to the directory /opt

2 .Uncompress the tar file

# tar -zxvf gamess-current.tar.tar

3. Go to the gamess directory

# cd gamess

4. Creating actvte.x file

# cd tools
# cp actvte.code actvte.f
# Replace all "*UNX" by " "(4 spaces with out " ") in the file actvte.f
# ifort -o actvte.x actvte.f
# rm actvte.f
# cd ..

5. Building the Distributed Data Interface(DDI) with OpenMPI:

# cd ddi
# vim compddi

 
5a. Editing the compddi file

## Set machine type (approximately line 18): ##
set TARGET=linux-ia64

## Set MPI communication layer (approximately line 48): ##
set COMM = mpi

## Set include directory for OpenMPI (approximately line 105): ##
## where is mpi header "mpi.h" is located ##
set MPI_INCLUDE_PATH = '-I/usr/mpi/intel/include'

5b. Compile compddi with OpenMPI

## Build DDI with OpenMPI ##
# ./compddi
# cd ..

If building completed successfully then library libddi.a will appear. Otherwise check compddi.log for errors.

6. Compiling the GAMESS:

6a. Editing file comp

vim comp
## Set machine type (approximately line 15): ##
set TARGET=linux-ia64

## Set the GAMESS root directory (approximately line 16): ##
chdir /opt/gamess

## Uncomment (approximately line 1461): ##
setenv MKL_SERIAL YES

6b Editing file compall

## Set machine type (approximately line 16): ##
set TARGET=linux-ia64

## Set the GAMESS root directory (approximately line 17): ##
chdir /opt/gamess

## Set to use Intel® C++ Compiler (approximately line 70): ##
if ($TARGET == linux-ia64) set CCOMP='icc'

6c Compiling the GAMESS:

# ./compall
# cd ..

7. Liniking the GAMESS with Intel® Software products:

7a Edit the file lked

## Set machine type (approximately line 18): ##
set TARGET=linux-ia64

## Set the GAMESS root directory (approximately line 19): ##
chdir /opt/games

## Check the  MKL environment (approximately line 511) is correct: for (x86_64)##
setenv  setenv MKLPATH `ls -d /opt/intel/mkl/*/lib/em64t`
set mklver=`ls /opt/intel/mkl`

## Set the message passing libraries in a single line (approximately line 710): ##
set MSG_LIBRARIES='../ddi/libddi.a -L/usr/local/lib -lmpi -lpthread'

7b Link the GAMESS

# ./lked

If linking completed successfully then executable file gamess.00.x will appear

8. Running the Application

8 Running the Application :
This section below describes how to execute GAMESS with Intel and OpenMPI. For further information check file ./ddi/readme.ddi.
For the testing GAMESS will be used script rungms as the base.

8a

## Set the target for execution to mpi (line 59): ##
set TARGET=mpi

## Set a directory SCR where large temporary files can reside(line 60): ##
set SCR=/scratch

## Correct the setting environment variables ERICFMT and MCPPATH (lines 127and 128): ##
setenv ERICFMT /opt/gamess/ericfmt.dat
setenv MCPPATH /opt/gamess/mcpdata

## Replace all “~$USER” by “/opt/gamess/tests”. Or by other directory. ##
## NOTE: Directory /scratch should exist. If no then create it. ##
## Replace all “/home/mike/gamess” by “/opt/gamess”. ##

## Correct the environment variables for Intel® MKL and OpenMPI (lines 948 and 953): ##
setenv LD_LIBRARY_PATH /opt/intel/mkl/10.2.4.032/lib/em64t $LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH /usr/local/lib:$LD_LIBRARY_PATH.

## Correct setting environment variables to execution OpenMPI path (line 954): ##
set path=(/usr/local/bin $path)

Now choose the testcase from directory ./tests and run GAMESS.
$./rungms exam08
The output data will be stored in the directory /scratch.

To execute GAMESS on 2 or more processes on 1 node:
$ ./rungms exam08 00 2