Using Ethtool to query Network and Driver Information

Ethtool is a utility for configuration of Network Interface Cards (NICs). This utility allows querying and changing settings such as speed, port, auto-negotiation, PCI locations and checksum offload on many network devices, especially Ethernet devices.

1. Query the specified network device for associated driver information

# ethtool -i ens3f1np1
driver: mlx5_core
version: 5.7-1.0.2
firmware-version: 16.34.1002 (MT_0000000416)
expansion-rom-version:
bus-info: 0000:0f:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

2. Enable an operator to easily identify the adapter by sight.
This involves blinking one or more LEDs on the specified network port.

# ethtool -p ens3f1np1 5

where integer 5 represents the time in seconds to perform the action,

3. Turn off the AutoNegotiation and fixed it at 25GB

ethtool -s ens3f1np1 --speed 25000 --autoneg off --duplex full

References:

Red Hat Documentation 11.8. Ethtool

Using firewall-cmd to configure gateways and isolated client network on CentOS-7 and Rocky Linux 8

Objectives:

Compute Nodes in an HPC environment are usually physically isolated from the public network and has to route through the gateway which are often found in Head Node or any delegated Node in small or small-medium size cluster to access the internet or to access company LAN to access LDAP, you can use the firewall-cmd to route the traffic through the interconnect facing the internet.

Scenario:

Traffic will be routed through the Head Node’s eno1 (internet facing) from the Head Node’s eno2 (private network). The interconnect eno1 is attached to a switch where the compute nodes are similarly attached. Some

  1. 192.168.1.0/24 is the private network subnet.
  2. 192.168.1.1 is the IP Address of the Head Node
  3. 155.1.1.2 is the IP Address of the external-facing ethernet ie eno1

Check the zones.

# firewall-cmd --list-all-zones

Check the Active Zones

# firewall-cmd --get-active-zones
external
  interfaces: eno2
internal
  interfaces: eno1

Enable masquerade at the Head Node’s External Zone

IP masquerading is a process where one computer acts as an IP gateway for a network. For masquerading, the gateway dynamically looks up the IP of the outgoing interface all the time and replaces the source address in the packets with this address.

You use masquerading if the IP of the outgoing interface can change. A typical use case for masquerading is if a router replaces the private IP addresses, which are not routed on the internet, with the public dynamic IP address of the outgoing interface on the router.

For more information. Do take a look at 5.10. Configuring IP Address Masquerading

# firewall-cmd --zone=external --query-masquerade 
no
# firewall-cmd --zone=external --add-masquerade --permanent
# firewall-cmd --reload

Compute Nodes at the Private Network 

(Assuming that eno1 is connected to the private switch). It is very important that you input the gateway at the compute node’s /etc/sysconfig/network-scripts/ifcfg-eno1)

.....
.....
DEVICE=enp47s0f1
ONBOOT=yes
IPADDR=192.168.1.2 #Internal IP Address of the Compute Node
NETMASK=255.255.255.0
GATEWAY=192.168.1.1 #Internal IP Address of the Head Node

Next, you have to put the Network Interface of the Client in the Internal Zone of the firewall-cmd. Assuming that eno1 is also used by the Client Network

# firewall-cmd --zone=internal --change-interface=eno1 --permanent

You may want to set the selinux to disabled

# setenforce 0

Configure the Head Node’s External Zone.

For Zoning, do take a look at 5.7.8. Using Zone Targets to Set Default Behavior for Incoming Traffic

For this setting, we have chosen target “default”

# firewall-cmd --zone=external --set-target=default

You can configure other settings. For the External Zone. For example, add SSH Service, mDNS

# firewall-cmd --permanent --zone=external --add-service=ssh
# firewall-cmd --permanent --zone=external --add-service=mdns
# firewall-cmd --runtime-to-permanent
# firewall-cmd --reload

Make sure the right Ethernet is placed in the right Zone. For External-Facing Ethernet Card, (eno2), you may want to place it

# firewall-cmd --zone=external --change-interface=eno2 --permanent

For Internal Facing Ethernet Card, (eno1), you want want to place it

# firewall-cmd --zone=internal --change-interface=eno1 --permanent

Configure the firewall-Source of Internal Network (eno1)

# firewall-cmd --zone=internal --add-source=192.168.1.0/24

Checking the Settings in the “firewall-cmd –get-active-zones”

# firewall-cmd --get-active-zones
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces: eno1
  sources: 192.168.1.0/32
  services: dhcpv6-client mdns ssh
  ports:
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eno2
  sources:
  services: dhcpv6-client ssh
  ports: 
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Check the Firewall Status

systemctl status firewalld.service

Compiling cmake-3.21.3 on Rocky Linux 8.5

If you are compiling the cmake-3.21.3 on Rocky Linux 8.5, and encounter the issues

-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR)

The problem is easy to resolve, you just need to install the openssl and openssl-devel libraries

# dnf install openssl openssl-devel

You should be able to bootrap easily.

Alternatively, instead of using ./boostrap, you can use the traditional configure command

#./configure --prefix=/usr/local/cmake-3.21.3
# make
# make install

“This system is not registered to Red Hat Insights” after log in to Rocky Linux 8.5

If you have warning messages such as after your login to your

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

To remove the warning permanently, you can do a

# dnf remove insights-client

References:

Installing MLNX_OFED 5.5-1 on Rocky Linux 8.5

If you are installing MLNX-OFED-5.5-1 on Rocky Linux 8.5, you may want to download the drivers from Nvidia Linux Drivers

Step 1: Installing Prerequistics

# dnf install tk tcsh tcl gcc-gfortran kernel-modules-extra

Step 2a: Installing MLNX on Rocky 8.5

If you just do a ./mlnxofedinstall

# ./mlnxofedinstall
Current operation system is not supported!

Step 2b: Force Install with the right distro.

# ./mlnxofedinstall --distro rhel8.5 --force
.....
.....
.....
Device #1:
----------

  Device Type:      ConnectX5
  Part Number:      MCX512F-ACH_Ax_Bx
  Description:      ConnectX-5 EN network interface card; with host management 25GbE Dual-port SFP28; PCIe3.0 x16; ROHS
  PSID:             MT_0000000416
  PCI Device Name:  10:00.0
  Base GUID:        xxxxxxxxxxxx
  Base MAC:         yyyyyyyyyyyy
  Versions:         Current        Available
     FW             16.31.1014     16.32.1010
     PXE            3.6.0403       3.6.0502
     UEFI           14.24.0013     14.25.0017

  Status:           Update required

After installing…….

Restart needed for updates to take effect.
Log File: /tmp/PAl8Z5mkHc
Real log file: /tmp/MLNX_OFED_LINUX.150443.logs/fw_update.log
To load the new driver, run:
/etc/init.d/openibd restart

Step 3: You have to remove and reload the drivers before you can do the /etc/init.d/openibd restart

[root@h00 media]# modprobe -rv ib_isert rpcrdma ib_srpt
rmmod ib_isert
rmmod iscsi_target_mod
rmmod rpcrdma
rmmod ib_srpt
[root@h00 media]# /etc/init.d/openibd restart
Unloading HCA driver:                                      [  OK  ]
Loading HCA driver and Access Layer:                       [  OK  ]

References:

  1. Driver Installation of Mellanox InfiniBand
  2. Mellanox NIC driver: Current Operation System is not supported

Disable the web console message from Cockpit on Rocky Linux 8.5

When you logged on to Rocky Linux 8.5, you may see this message.

Activate the web console with: systemctl enable --now cockpit.socket

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

If you wish to disable the message, you may want to

rm -f /etc/motd.d/cockpit

The command will remove the symlink to file /run/cockpit/motd

Setting up NTP Client in Rocky Linux 8.5

Prerequisites Step 1: Endure you are in the correct time zone

# timedatectl
               Local time: Wed 2022-04-20 10:04:44 +08
           Universal time: Wed 2022-04-20 02:04:44 UTC
                 RTC time: Wed 2022-04-20 02:04:44
                Time zone: Asia/Singapore (+08, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

Prerequisites Step 2: List Time Zone

# timedatectl list-timezones
.....
Asia/Singapore
.....

Prerequisites Step 3: Set Time Zone

# timedatectl set-timezone Asia/Singapore

In Rocky Linux 8.5, the ntp package is no longer supported and it is implemented by the chronyd (a daemon that runs in user-space) which is provided in the chrony package.

chrony works both as an NTP server and as an NTP client, which is used to synchronize the system clock with NTP servers.

To install the chrony suite, use the DNF Package Manager.

# dnf install chrony

Enable the Service

# systemctl start chronyd
# systemctl status chronyd
# systemctl enable chronyd

Check it is synchronised

[root@h00 etc]# timedatectl
               Local time: Wed 2022-04-20 10:19:56 +08
           Universal time: Wed 2022-04-20 02:19:56 UTC
                 RTC time: Wed 2022-04-20 02:19:56
                Time zone: Asia/Singapore (+08, +0800)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Setting up NTP Client Using Chrony in Rocky Linux 8.5

# vim /etc/chrony.conf
.....
pool sg.pool.ntp.org iburst
.....
# systemctl restart chronyd

Show the current time sources that chronyd is accessing

# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 178.128.223.142               0   6     0     -     +0ns[   +0ns] +/-    0ns
.....
.....
.....

References:

Comparison between the /etc/os-release of RHEL-8.5 and Rocky Linux 8.5

Rocky Linux is a production-ready downstream version of Red Hat Enterprise Linux started by original founder of CentOS, Gregory Kurtzer. the OS is almost identical under intensive development by the community.

For RHEL-8.5,

NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Green Obsidian)"
ID="rhel"
ID_LIKE=”fedora"
VERSION_ID=”8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Green Obsidian)"
ANSI_COLOR="0;31"
CPE_NAME=”cpe:/o:redhat:enterprise_linux:8.5:GA”
HOME_URL=”https://www.redhat.com/”
DOCUMENTATION_URL=”https://access.redhat.com/documentation/red_hat_enterprise_linux/8/”
BUG_REPORT_URL=”https://bugzilla.redhat.com/”

For Rocky-Linux-8.5,

NAME="Rocky Linux"
VERSION="8.5 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.5 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8.5:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"

Finding physical cpus, cores and logical cpus

Number of Active Physical Processors

% grep physical.id /proc/cpuinfo | sort -u | wc -l
8

Number of cores per CPU

% grep cpu.cores /proc/cpuinfo | sort -u
cpu cores       : 26

Therefore the total number of cores is
8×26 = 208 cores

Number of Logical Processors

The number of cores seen by the Linux. Since the Server has switched on Hyperthreading

% grep processor /proc/cpuinfo | wc -l
416

References:

  1. How to find the number of physical cpus, cpu cores, and logical cpus