Installing and using Mellanox HPC-X Software Toolkit


Taken from Mellanox HPC-X Software Toolkit User Manual 2.3

Mellanox HPC-X is a comprehensive software package that includes MPI and SHMEM communication libraries. HPC-X includes various acceleration packages to improve both the performance and scalability of applications running on top of these libraries, including UCX (Unified Communication X) and MXM (Mellanox Messaging), which accelerate the underlying send/receive (or put/get) messages. It also includes FCA (Fabric Collectives Accelerations), which accelerates the underlying collective operations used by the MPI/PGAS languages.



% tar -xvf hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-5.0-
% cd hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-5.0-
% export HPCX_HOME=/usr/local/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-5.0-

Loading HPC-X Environment from BASH

HPC-X includes Open MPI v4.0.x. Each Open MPI version has its own module file which can be used to load the desired version

% source $HPCX_HOME/
% hpcx_load
% env | grep HPCX
% mpicc $HPCX_MPI_TESTS_DIR/examples/hello_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_c
% mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c
% oshcc $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c.c -o $HPCX_MPI_TESTS_DIR/examples/
% hello_oshmem_c
% oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c
% hpcx_unload

Loading HPC-X Environment from Modules

You can use the already built module files in hpcx.

% module use $HPCX_HOME/modulefiles
% module load hpcx
% mpicc $HPCX_MPI_TESTS_DIR/examples/hello_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_c
% mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c
% oshcc $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c.c -o $HPCX_MPI_TESTS_DIR/examples/
% oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c
% module unload hpcx

Building HPC-X with the Intel Compiler Suite

Do take a look at the Mellanox HPC-X® ScalableHPC Software Toolkit


  1. Mellanox HPC-X Software Toolkit User Manual 2.3
  2. Mellanox HPC-X® ScalableHPC Software Toolkit

Fabric Debug Initiation using ibdiagnet (Part 1)

Learn some of these steps from Mellanox Academy Online Training

Step 1: Clear all counters and begin the test execution

ibdiagnet -pc

Wait for a while. Usually, it may take 30 to 60 mins

Check for errors that exceed the allowed threshold

ibdiagnet -ls 25 -lw 4x -P all=1 --pm_pause_time 30
  • Specify the link speed
    -ls <2.5|5|10|14|25|50> 
  • Specify the Link width
    -lw <1x|4x|8x|12x>
  • Check Information provide from all counters and display each one of them crossing threshold of 1
    -P all=1
  • The time between the two samples is set by the –pm_pause_time option

Webinar – Build the Most Powerful Data Center with GPU Computing Technology and High-speed Interconnect

Build the Most Powerful Data Center with GPU Computing Technology and High-speed Interconnect

Date: Thursday, June 11, 2020
Time: 11:00am-12:30pm Singapore Time

Register here 

Please join NVIDIA as we discuss how to design a well-balanced system that maximizes performance and scalability of various workloads using NVIDIA GPUs and interconnect

Speakers will provide an overview of the state-of-the-art NVIDIA GPU accelerated compute architecture and In-Network computing fabric and how they come together with one goal: to deliver a solution that democratizes supercomputing power, making it readily accessible, installable, and manageable in a modern business setting. To learn more about this webinar click here

Mellanox Introduces Advanced Network Telemetry Technology

Mellanox today announced an advanced Network Telemetry technology that provides visibility into Data Plane Anomalies – for public cloud, private cloud and enterprise computing.

What Just Happened (WJH) is a new intelligent monitoring technology that goes well beyond conventional streaming telemetry and Inband Network Telemetry (INT) offerings. These conventional telemetry offerings generate enormous amounts of data that is mined from network port counters and statistical packet sampling and hop-by-hop INT data. This type of telemetry data lacks any context or actionable details on abnormal network behavior, forcing the network admin or monitoring tool to perform time consuming analysis to find the root cause of network issues. WJH event-based telemetry delivers direct and actionable information and thus eliminates the guesswork from network troubleshooting.

In Mellanox based switch systems, WJH inspects packets across all ports at line-rate, at speeds that would overwhelm traditional Deep Packet Inspection solutions. WJH then identifies anomalous behavior, consolidates this into concise, specific, and actionable data that enables rapid remediation and recovery of cloud applications and critical business operations.

How Mellanox ConnectX and BlueField Empower Cyber Security Solutions

News from Mellanox

  1. Mellanox Turns Zero-Trust to HERO-Trust
    A Mellanox blog discussing the zero-trust security model, and how Mellanox BlueField SmartNIC empowers zero-trust security solutions
  2. ConnectX & BlueField Empower Cloud Security SolutionsEye on Mellanox YouTube VideoA short video clip discussing how ConnectX & BlueField empower cloud security solutions
  3. Guardicore and Mellanox Deliver Agentless and High-Performance Micro-Segmentation in Hybrid Cloud
    Guardicore is a cyber security solution provider which offers a complete micro-segmentation solution for enforcing security policies in data-center networks. Micro-segmentation is considered a priority for CISO’s in 2019 and beyond. We have recently partnered with Guardicore to have their software run on top of BlueField, enabling agentless and high-performant micro-segmentation solution. The attached solution brief describes the key benefits of applying BlueField for micro-segmentation and key benefits of the joint solution. 

White Paper on Mellanox seamless integration with OpenStack

Mellanox’s Ethernet solutions enable cloud data centers to achieve the highest efficiency through a high-performance, low latency cloud network with rich network offload with acceleration and automation features.
Check out the attached Whitepaper from Mellanox and see how we can help you get it there!

See Mellanox Seamlessly Integrates with OpenStack, Increasing Efficiency and Reducing Operational Costs


Mellanox SN2100 Ethernet Bundle Configurator

Mellanox has a Ethernet Configurator which you can even use for other model.

Here’s how it works:
Choose the following:
1. Choose the setup

  • 1 switch
  • 2 switches connected with mLAG
  • 5 switches: testing both mLAG as well as leaf spine (BGP/ECMP) – simple L3/VTEP/VXLAN Routing (VXLAN on Cumulus only for now)

2. Choose the speed of the host/NIC

  • 10/25/100GbE

3. Choose the operating system

  • ONYX
  • Cumulus Linux

What you get is:
1. Topology: how to connect, which specific ports should be used

2. Solution brief for the selected scenario

3. One Click Configuration files:

  • For the CLI guys: running config
  • For the Ansible guys: Ansible playbooks (ONYX Ansible for mLAG and L3 will be added soon)