This paper supports this conclusion with three real application benchmarks running on IBM’s Rackswitch G8316, a 40Gb Ethernet aggregation switch, in conjunction with Chelsio Communications’ 40Gb Ethernet Unified Wire network adapter. This paper shows how iWARP offers comparable application level performance at 40Gbps with the latest InfiniBand FDR speeds.
Chelsio
Compiling Chelseio IWARP Drivers (2.8.0.0) on CentOS 5
The below is a subset of the Chelsio 2.8.0.0 ReadMe
The Chelsio Unified Wire software has been developed to run on 64-bit Linux
based platforms. Following is the list of Drivers/Software and supported Linux
distributions. Here is a subset of the README.
The OS I used was CentOS 5.8
|########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL5.8,2.6.18-308.el5 |NIC/TOE,vNIC,iWARP,WD-UDP*,WD-TOE*,iSCSI Target*, | | |Bonding,IPv6,Bypass*,Sniffer & Tracer | | |UM(Agent,Client),UDP-SO,Filtering,TM | |------------------------|-----------------------------------------------------| |RHEL5.9,2.6.18-348.el5 |NIC/TOE*,vNIC*,iWARP*,WD-UDP*,WD-TOE*,iSCSI Target*, | | |Bonding*,IPv6*,Bypass*,Sniffer & Tracer*,UDP-SO*, | | |Filtering*,TM* | |------------------------|-----------------------------------------------------| |RHEL6.3, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE*,iSCSI Target*, | |2.6.32-279.el6 |iSCSI Initiator*,FCoE Initiator*, | | |Bonding,IPv6,Bypass*,Sniffer & Tracer,UDP-SO, | | |UM(Agent,Client,WebGUI),Filtering,TM | |------------------------|-----------------------------------------------------| |RHEL6.4, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |2.6.32-358.el6 |iSCSI Initiator,FCoE Initiator,Bonding,IPv6,Bypass, | | |Sniffer & Tracer,UDP-SO,UM(Agent,Client,WebGUI), | | |Filtering,TM,uBoot(DUD) | |------------------------|-----------------------------------------------------|
Strangely, I was not able to compile with 3.5.1. It seems that the compat-rdma on 3.5.1 is having issues with CentOS 5.8. See Failed to build compat-rdma RPM when compiling OFED 3.5.1 on CentOS 5.8
I tried with OFED 1.5.4.1, but errors occurred as well. But compiling OFED 1.5.3.2 works well and Chelsio T420-BCH was able to compile nicely with OFED 1.5.3.2. To download OFED 1.5.3.2, do visit the OFED Downloads Site
Part 1
To compile from source
i. Download the tarball ChelsioUwire-x.x.x.x.tar.gz
ii. Untar the tarball
[root@host]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package
directory. Build the source:
[root@host]# make
iv. Install the drivers, tools and libraries:
[root@host]# make install
v. The default configuration tuning option is Unified Wire.
The configuration tuning can be selected using the following commands:
[root@host]# make CONF=(T5/T4 Configuration)
[root@host]# make CONF=(T5/T4 Configuration install)
(where T5/T4 Configuration is
UNIFIED_WIRE, HIGH_CAPACITY_TOE, HIGH_CAPACITY_RDMA, LOW_LATENCY, UDP_OFFLOAD, T5_WIRE_DIRECT_LATENCY)
Part 2 – Installing Individual Drivers
i. To build and install iWARP driver against outbox OFED:
[root@host]# make iwarp
[root@host]# make iwarp_install
Part 3a – Loading IWARP Drivers
Manually Load Drivers
To load the iWARP driver we need to load the NIC driver & core RDMA drivers first:
[root@host]# modprobe cxgb4
[root@host]# modprobe iw_cxgb4
[root@host]# modprobe rdma_ucm
Part 3b – Automatic IWARP Drivers
To load the Chelsio iWARP drivers automatically, add this additional lines to /etc/modprobe.conf
options iw_cxgb4 peer2peer=1 install cxgb4 /sbin/modprobe -i cxgb4; /sbin/modprobe -f iw_cxgb4; /sbin/modprobe rdma_ucm alias eth1 cxgb4 # assuming eth1 is used by the Chelsio interface
Finally Reboot the system to load the new modules
References:
PBS scripts for mpirun parameters for Chelsio / Infiniband Cards
If you are running Chelsio Cards, you may want to specify the mpirun parameters to ensure the
/usr/mpi/intel/openmpi-1.4.3/bin/mpirun -mca btl openib,sm,self --bind-to-core --report-bindings -np $NCPUS -machinefile $PBS_NODEFILE $PBS_O_WORKDIR/$file
–bind-to-core: Bind each MPI process to a core
–mca btl openib,sm,self: (Infiniband, shared memory, the loopback)
For information on Interprocess communication with shared memory,
Installing Chelsio driver CD on an ESX 4.x host
This article is taken and modified from Installing the VMware ESX/ESXi 4.x driver CD on an ESX 4.x host (VMware Knowledge Base)
Step 1: Download the Chelsio Drivers for ESX
Download from relevant drivers for your sepcific cards from Chelsio Download Centre
Step 2: Follow the instruction from VMware
-
Download the driver CD from the vSphere Download Center.
-
Extract the ISO on your local workstation using an third-party ISO reader (such as WinISO). Alternatively, you can mount the ISO via SSH with the command:
mkdir /mnt/iso mount -o loop filename.iso /mnt/iso
Note: Microsoft operating systems after Windows Vista include a built-in ISO reader.
- Use the Data Browser in the vSphere Client to upload the ZIP file that was extracted from the ISO to your ESX host.
Alternatively, you can use a program like WinSCP to upload the file directly to your ESX host. However, you require root privileges to the host to perform the upload.
-
Log in to the ESX host as root directly from the Service Console or through an SSH client such as Putty.
-
Place the ESX host in Maintenance Mode from the vSphere Client.
-
Run this command from the Service Console or your SSH Client to install the bundled package:
esxupdate –bundle=<name of bundled zip> update
-
When the package has been installed, reboot the ESX host by typing reboot from the Service Console.
Note: VMware does not endorse or recommend any particular third party utility, nor are the above suggestions meant to be exhaustive.
Installing Chelsio Unified Wire from RPM for CentOS 5
This writeup is taken from the ChelsioT4 UnifiedWire Linux UserGuide (pdf) and trimmed for installation on RHEL5.4 or CentOS 5.4. But it should apply for other RHEL / CentOS versions
Installing Chelsio Software
1. Download the tarball specific to your operating system and architecture from our Software download site http://service.chelsio.com/
2. For RHEL 5.4, untar using the following command:
# tar -zxvf ChelsioUwire-1.1.0.10-RHEL-5.4-x86_64.tar.gz
3. Navigate to “ChelsioUwire-x.x.x.x” directory. Run the following command
# ./install.sh
4. Select „1‟ to install all Chelsio modules built against inbox OFED or select „2‟ to install OFED-1.5.3 and all Chelsio modules built against OFED-1.5.3.
5. Reboot system for changes to take effect.
6. Configure the network interface at /etc/sysconfig/network-scripts/ifcfg-ethx
Compiling and Loading of iWARP (RDMA) driver
To use the iWARP functionality with Chelsio adapters, user needs to install the iWARP drivers as well as the libcxgb4, libibverbs, and librdmacm libraries. Chelsio provides the iWARP drivers and libcxgb4 library as part of the driver package. The other libraries are provided as part of the Open Fabrics Enterprise Distribution (OFED) package.
# modprobe cxgb4 # modprobe iw_cxgb4 # modprobe rdma_ucm
# echo 1 >/sys/module/iw_cxgb4/parameters/peer2peer
Testing connectivity with ping and rping.
On the Server,
# rping -s -a server_ip_addr -p 9999
On the Client Server,
# rping -c –Vv -C10 -a server_ip_addr -p 9999
You should see ping data like this
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT...
Configuring VMDirectPath I/O pass-through devices on an ESX host with Chelsio T4 Card (Part 2)
Note for Installation of the VM:
Remember to add the PCI/PCIe Device to the VM. Upon adding, you should be able to see the “10:00.4 | Chelsio Communications Chelsio T4 10GB Ethernet”. See above Pix
Proceed with installation of the VM, you should be able to see the Ethernet settings. Do proceed with the installation of OFED and Chelsio Drivers.
Information:
Configuring VMDirectPath I/O pass-through devices on an ESX host with Chelsio T4 Card (Part 1)
For Part 1, the article is taken from Configuring VMDirectPath I/O pass-through devices on an ESX host. Part (2), we will deal with Chelsio T4 Card configuration after the Passthrough has been configured.
1. Configuring pass-through devices
- Select an ESX host from the Inventory of VMware vSphere Client.
- On the Configuration tab, click Advanced Settings. The Pass-through Configurationpage lists all available pass-through devices.Note:A green icon indicates that a device is enabled and active.An orange icon indicates that the state of the device has changed and the host must be rebooted before the device can be used.
- Click Edit.
- Select the devices and click OK.Note: If you have a chipset with VT-d, when you click Advanced Settings in vSphere Client, you can select what devices are dedicated to the VMDirectPath I/O.
- When the devices are selected, they are marked with an orange icon. Reboot for the change to take effect. After rebooting, the devices are marked with a green icon and are enabled.Note:The configuration changes are saved in the /etc/vmware/esx.conf file. The parent PCI bridge, and if two devices are under the same PCI bridge, only one entry is recorded.The PCI slot number where the device was connected is 00:0b:0. It is recorded as:/device/000:11.0/owner = “passthru”Note: 11 is the decimal equivalent of the hexadecimal 0b.
2. To configure a PCI device on a virtual machine:
- From the Inventory in vSphere Client, right-click the virtual machine and choose Edit Settings.
- Click the Hardware tab.
- Click Add.
- Choose the PCI Device.
- Click Next.Note: When the device is assigned, the virtual machine must have a memory reservation for the full configured memory size.
3. Information