Switching between Ethernet and Infiniband using Virtual Protocol Interconnect (VPI)


According to the Open Fabrics Alliance Documents, Open Fabrics Enterprise Distribution (OFED) ConnectX driver (mlx4) in OFED 1.4 Release Notes.

It is recommended to use the QSA Adapter (QSFP+ to SFP+ adapter) which is the world’s first solution for the QSFP to SFP+ conversion challenge for 40GB/Infiniband to 10G/1G. For more information, see Quad to Serial Small Form Factor Pluggable (QSA) Adapter to allow for the hardware

gfx_01604

Here is the summary of the excerpts from the document.

Overview
mlx4 is the low level driver implementation for the ConnectX adapters designed by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter, as an Ethernet NIC, or as a Fibre Channel HBA. The driver in OFED 1.4 supports Infiniband and Ethernet NIC configurations. To accommodate the supported configurations, the driver is split into three modules:

  1. mlx4_core
    Handles low-level functions like device initialization and firmware commands processing. Also controls resource allocation so that the InfiniBand and Ethernet functions can share the device without interfering with each other.
  2. mlx4_ib
    Handles InfiniBand-specific functions and plugs into the InfiniBand midlayer
  3. mlx4_en
    A new 10G driver named mlx4_en was added to drivers/net/mlx4. It handles Ethernet specific functions and plugs into the netdev mid-layer.

Using Virtual Protocol Interconnect (VPI) to switch between Ethernet and Infiniband

Loading Drivers

    1. The VPI driver is a combination of the Mellanox ConnectX HCA Ethernet and Infiniband drivers. It supplies the user with the ability to run Infiniband and Ethernet protocols on the same HCA.
    2. Check the MLX4 Driver is loaded, ensure that the
      # vim /etc/infiniband/openib.conf
      # Load MLX4_EN module
      MLX4_EN_LOAD=yes
    3. If the MLX4_EN_LOAD=no, the Ethernet Driver can be loaded by running
      # /sbin/modprobe mlx4_en

Port Management / Driver Switching

  1. Show Port Configuration
    # /sbin/connectx_port_config -s
    --------------------------------
    Port configuration for PCI device: 0000:16:00.0 is:
    eth
    eth
    --------------------------------
  2. Looking at saved configuration
    # vim /etc/infiniband/connectx.conf
  3. Switching between Ethernet and Infiniband
    # /sbin/connectx_port_config
  4. Configuration supported by VPI
    - The following configurations are supported by VPI:
    	Port1 = eth   Port2 = eth
    	Port1 = ib    Port2 = ib
    	Port1 = auto  Port2 = auto
    	Port1 = ib    Port2 = eth
    	Port1 = ib    Port2 = auto
    	Port1 = auto  Port2 = eth
    
      Note: the following options are not supported:
    	Port1 = eth   Port2 = ib
    	Port1 = eth   Port2 = auto
    	Port1 = auto  Port2 = ib

For more information, see

  1. ConnectX -3 VPI Single and Dual QSFP+ Port Adapter Card User Manual (pdf)
  2. Open Fabrics Enterprise Distribution (OFED) ConnectX driver (mlx4) in OFED 1.4 Release Notes
Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.