Most of this material for this blog entry is taken the documentation Guide named Chelsio iWARP installation and setup guide (pdf) . This Blog Entry “Chelsio iWARP Installation and Setup Guide for CentOS 5.4” is an modification from a user’s perpective of the original document.
1. Install RPMForge first
You will need some utilities from rpmforge to install iWARP successfully. For more information o install RPMForge, see Installing RPMForge (Linux Toolkits)
2. Yum Install the following utilities which is required for the iWARP Installation
# yum install libevent-devel nfs-utils-lib-devel tcl-devel
3. Download the latest package that matched your Chelsio Network Adapters from Open Fabrics Alliance. Here is the latest OFED Package Download Site
4. Unpacked and install the OFED Drivers
# wget http://69.55.239.13/downloads/OFED/ofed-1.5.1/OFED-1.5.1.tgz
# tar -zxvf OFED-1.5.1.tgz
# cd OFED-1.5.1
# ./install.pl
a. Inside the menu
1. Choose option 2 to install OFED package.
2. Then choose option 3 to install all OFED libraries.
3. Then choose default options in which come while executing ./install.pl script
to build and install OFED OR
4. If you are familiar with OFED installation you can choose option 2 then option 4 for
customized installation.
b. If you encounter error like
file /lib/modules/2.6.18-164.el5/updates/kernel/drivers/net/cxgb3/cxgb3.ko from install of
kernel-ib-1.5.1-2.6.18_164.el5.x86_64 conflicts with file from package
cxgb3toe-1.4.1.2-custom.x86_64
*It is likely that you use the cxgb3toe-1.4.1.2-custom.x86_64.rpm to install the drivers. This immediately conflicts with kernel-ib-1.5.1-2.6.18_164.el5.x86_64.rpm. It is advisisable to install using make && make install. See Installing Chelsio 10GE Driver on CentOS 5.4
c. Resolution for the error above problem
# rpm -e cxgb3toe-1.4.1.2-custom.x86_64.rpm
* Start from Step 4 and do the ./install.pl again.
5. After installation reboot system for changes to take effect.
6. Set Chelsio driver option for MPI connection changes.
Give the below command on all systems
# echo 1 > /sys/module/iw_cxgb3/parameters/peer2peer
OR to make it permanent, add the following line to /etc/modprobe.conf to set the option at module load time:
options iw_cxgb3 peer2peer=1
*The option setting in file /etc/modprobe.conf shall take effect upon system reboot
7. Checking Chelsio iWARP Drivers compatibility with Chrlsio Linux Drivers. There is a whole list as shown in Chelsio iWARP Drivers compatibility with Chelsio Linux drivers. Do take a good look
| OFED Package |
Cxgb3toe-W.X.YY.ZZZ driver |
Firmware |
Supported/Not Supported/Not Tested |
| OFED-1.5.1 |
Cxgb3toe-1.4.1.2 |
7.10.0 |
Not Supported |
| OFED-1.5.1 |
Cxgb3toe-1.4.1.2 |
7.8.0 |
Supported |
| OFED-1.5.1 |
Cxgb3toe-1.4.1.2 |
7.4.0 |
Not Supported |
8. Installing Chelsio cxgb3toe-W.X.YY.ZZZ driver with OFED-X.Y.Z package. Do follow the blog entry for
a. Check Blog Entry on Chelsio iWARP Drivers compatibility with Chelsio Linux drivers. You may not need to do the Performance Tuning.
9. To load the Chelsio iWARP drivers on RHEL 5.4 or CentOS 5.4, add this additional lines to /etc/modprobe.conf
options iw_cxgb3 peer2peer=1
install cxgb3 /sbin/modprobe -i cxgb3; /sbin/modprobe -f iw_cxgb3; /sbin/modprobe rdma_ucm
alias eth1 cxgb3 # assuming eth1 is used by the Chelsio interface
10. Reboot the system to load the new modules
11. After rebooting, you should be have loaded iw_cxgb3 and rdma_ucm module, you should be able to see the ethernet interface(s) for the T3 device. Do configure them with the appropriate ip addresses, netmask etc.
a. Test I: Test Ping
After setting the ipaddress, netmask, gateway etc, you should be able to ping the ethernet interface.
b. Test IIa: Test RDMA (Server Preparation)
To test RDMA, use the rping command that is included in the librdmacm-utils rpm
On the server machine:
# rping -s -a server_ip_address -p 9999
* The server will be “waiting mode” for the client connection
c. Test IIb: Test RDMA (Client Preparation)
You have to setup the clients from Pt 1 to Pt 10 again. If you are using xcat, you may wish to use it to automate the setup of the client.
# rping -c –Vv -C10 -a server_ip_addr -p 9999
* You should see ping data like this on the client
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...
12. Great you are done. Read more on how to enable and compile with MPI and iWARP
The Linux Cluster 