Day: March 26, 2012

by

Installing Chelsio Unified Wire from RPM for CentOS 5

This writeup is taken from the ChelsioT4 UnifiedWire Linux UserGuide (pdf) and trimmed for installation on RHEL5.4 or CentOS 5.4. But it should apply for other RHEL / CentOS versions

Installing Chelsio Software

1. Download the tarball specific to your operating system and architecture from our Software download site http://service.chelsio.com/

2. For RHEL 5.4, untar using the following command:

# tar -zxvf ChelsioUwire-1.1.0.10-RHEL-5.4-x86_64.tar.gz

3. Navigate to “ChelsioUwire-x.x.x.x” directory. Run the following command

# ./install.sh

4. Select „1‟ to install all Chelsio modules built against inbox OFED or select „2‟ to install OFED-1.5.3 and all Chelsio modules built against OFED-1.5.3.

5. Reboot system for changes to take effect.

6. Configure the network interface at /etc/sysconfig/network-scripts/ifcfg-ethx

Compiling and Loading of iWARP (RDMA) driver

To use the iWARP functionality with Chelsio adapters, user needs to   install the iWARP drivers as well as the libcxgb4, libibverbs, and librdmacm   libraries. Chelsio provides the iWARP drivers and libcxgb4 library as part of the driver package. The other libraries are provided as part of the Open   Fabrics Enterprise Distribution (OFED) package.

# modprobe cxgb4
# modprobe iw_cxgb4
# modprobe rdma_ucm
# echo 1 >/sys/module/iw_cxgb4/parameters/peer2peer

Testing connectivity with ping and rping.

On the Server,

# rping -s -a server_ip_addr -p 9999

On the Client Server,

# rping -c –Vv -C10 -a server_ip_addr -p 9999

You should see ping data like this

ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs 
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst 
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu 
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv 
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw 
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx 
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy 
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz 
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA 
client DISCONNECT EVENT...
by

Configuring VMDirectPath I/O pass-through devices on an ESX host with Chelsio T4 Card (Part 2)

Note for Installation of the VM:

Remember to add the PCI/PCIe Device to the VM. Upon adding, you should be able to see the “10:00.4  | Chelsio Communications Chelsio T4 10GB Ethernet”. See above Pix

Proceed with installation of the VM, you should be able to see the Ethernet settings. Do proceed with the installation of OFED and Chelsio Drivers.

Information:

  1. Configuring VMDirectPath I/O pass-through devices on an ESX host with Chelsio T4 Card (Part 1)
by

Configuring VMDirectPath I/O pass-through devices on an ESX host with Chelsio T4 Card (Part 1)

For Part 1, the article is taken from Configuring VMDirectPath I/O pass-through devices on an ESX host. Part (2), we will deal with Chelsio T4 Card configuration after the Passthrough has been configured.

1. Configuring pass-through devices

To configure pass-through devices on an ESX host:
  1. Select an ESX host from the Inventory of VMware vSphere Client.
  2. On the Configuration tab, click Advanced Settings. The Pass-through Configurationpage lists all available pass-through devices.Note:A green icon indicates that a device is enabled and active.An orange icon indicates that the state of the device has changed and the host must be rebooted before the device can be used.
  3. Click Edit.
  4. Select the devices and click OK.Note: If you have a chipset with VT-d, when you click Advanced Settings in vSphere Client, you can select what devices are dedicated to the VMDirectPath I/O.
  5. When the devices are selected, they are marked with an orange icon. Reboot for the change to take effect. After rebooting, the devices are marked with a green icon and are enabled.Note:The configuration changes are saved in the /etc/vmware/esx.conf file. The parent PCI bridge, and if two devices are under the same PCI bridge, only one entry is recorded.The PCI slot number where the device was connected is 00:0b:0. It is recorded as:/device/000:11.0/owner = “passthru”Note: 11 is the decimal equivalent of the hexadecimal 0b.

2. To configure a PCI device on a virtual machine:

  1. From the Inventory in vSphere Client, right-click the virtual machine and choose Edit Settings.
  2. Click the Hardware tab.
  3. Click Add.
  4. Choose the PCI Device.
  5. Click Next.Note: When the device is assigned, the virtual machine must have a memory reservation for the full configured memory size.

 

3. Information

  1. Configuring VMDirectPath I/O pass-through devices on an ESX host with Chelsio T4 Card (Part 2)