Chelsio is one of the leaders for High Speed, Low Latency 10GE Adapters. On great interest to me is the TCP offloading and the iWARP capability of the Card. To complement this high-end quality cards, you have to use good quality high-end but very low latency from Blade Network Technologies (BNT)
1. Documentation (Impt)
- The various documentation list for Chelsio OEM-IBM Information can be found at my Linux Toolkit Blog.
- The most critical page is to go to the Chelsio Drivers Downloads page for a variety of OS can be found.
2. Installing the Drivers on CentOS 5.4
1. I have downgraded the OS from CentOS 5.5 to Centos 5.4 as the offloading bonding is not supported by latest latest CentOS 5.5 kernel
2. Download the Linux drivers under Terminator 3 (T3) family Ethernet Adapter drivers. At this point in writing, the latest Linux driver is cxgb3toe-1.4.1.2.tar.gz (TOE/NIC)
3. Untar the drivers and go into the directory
# tar -zxvf cxgb3toe-1.4.1.2.tar.gz # cd cxgb3toe-1.4.1.2
4. Follow the instruction on the READ ME found at source directory ( $SOURCE_DIR/cxgb3toe-1.4.1.2)
5. The src directory contains the driver source files for building kernel modules. To build the TOE driver, change to the src/ directory and run:
make make install
6. Go to the $SOURCE_DIR/cxgb3toe-1.4.1.2 directory/tools directory and copy the ifup-local and perftune.sh to the /sbin directory and perftune.sh will be run each time the interface is enabled.
# cd /tmp/cxgb3toe-1.4.1.2 directory/tools # cp ifup-tools /sbin # cp perftune.sh /sbin
7. Run the performance Tuning
/sbin/perftune.sh
You should see output something like this.
/ > Extract internal utility to /var/tmp/mmapr64. [ PASS ] > IRQ Balance daemon is not running. [ PASS ] > eth1: PCI-E x8 device using all lanes. [ PASS ] > eth1: Set IRQ 51 smp_affinity to CPU0. [ PASS ] > eth1: Set IRQ 59 smp_affinity to CPU1. [ PASS ] > eth1: Set IRQ 67 smp_affinity to CPU2. [ PASS ] > eth1: Set IRQ 202 smp_affinity to CPU3. [ PASS ] > eth1: Set IRQ 210 smp_affinity to CPU4. [ PASS ] > eth1: Set IRQ 218 smp_affinity to CPU5. [ PASS ] > eth1: Set IRQ 226 smp_affinity to CPU6. [ PASS ] > eth1: Set IRQ 234 smp_affinity to CPU7. [ PASS ] > TOM(toe0): Enable DDP. [ PASS ] > TOM(toe0): Set 'delayed_ack=2'. [ PASS ] > TOM(toe0)[eth4]: Disable TCP timestamps. [ PASS ] [ Set sysctls... ] > Set net.core.wmem_max="16777216" [ PASS ] > Set net.core.rmem_max="16777216" [ PASS ] > Set net.ipv4.tcp_timestamps="0" [ PASS ] > Set net.ipv4.tcp_rmem="4096 262144 16777216" [ PASS ] > Set net.ipv4.tcp_wmem="4096 262144 16777216" [ PASS ] > Set net.core.optmem_max="524288" [ PASS ] > Set net.core.netdev_max_backlog="200000" [ PASS ] [ System tuning is complete. ]
8. Configuring iWARP….Coming your way
The Linux Cluster 