Network

by

Understanding basic nmcli in Rocky Linux 9

In Rocky Linux 9, the nmcli command-line tool (Network Manager Command Line) replaces the traditional ifcfg files that we have been using since Rocky Linux 8. If you can Google “Why nmcli is replacing the ifcfg”, you will find a comprehensive list of key reasons why the transition took place. One thing that I like best is this particular answer

nmcli commands are designed to be easily automated and scripted (e.g., using Ansible), offering better control and error checking (syntax validation) compared to generating flat text files through scripts.

Usage 1a: List the NetworkManager connection profiles

# nmcli con
NAME   UUID                                  TYPE      DEVICE 
ens33  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ethernet  ens33  
lo     yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy  loopback  lo

Usage 1b: List the Network Devices and their status

# nmcli dev
DEVICE  TYPE      STATE                   CONNECTION 
ens33   ethernet  connected               ens33      
lo      loopback  connected (externally)  lo        

Usage 2a: Disable the connection of ens33

# nmcli con down ens33
Connection 'ens33' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)

Usage 2b: Enable the connection of ens33

# nmcli con up ens33
Connection 'ens33' successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)

Usage 2c: Show Connection Details

# nmcli con show ens33
[root@hpc-wfly-rl9 ~]# nmcli con show ens33
connection.id:                          ens33
connection.uuid:                        817c4ac5-49f4-3752-9a16-9d7460bed1c9
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              ens33
connection.autoconnect:                 yes
connection.autoconnect-priority:        -999
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1763952141
connection.permissions:                 --
connection.zone:                        --
connection.controller:                  --
connection.master:                      --
connection.slave-type:                  --
connection.port-type:                   --
connection.autoconnect-slaves:          -1 (default)
connection.autoconnect-ports:           -1 (default)
connection.down-on-poweroff:            -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.ip-ping-timeout:             0
connection.ip-ping-addresses:           --
connection.ip-ping-addresses-require-all:-1 (default)
connection.metered:                     unknown
connection.lldp:                        default
.....
.....

Usage 3: Set the static IP Address of the Ethernet Connection

# nmcli con mod ens33 ipv4.method manual ipv4.address 10.10.1.2/24 ipv4.gateway 10.10.1.1
# nmcli con up ens33

Usage 4a: Using conn to update DNS (replace manual scripting of /etc/resolv.conf)

# nmcli con mod ens33 ipv4.dns '8.8.8.8,8.8.8.4' 
# nmcli con show |grep dns
# nmcli con up ens33

At /etc/resolv.conf, you will notice

# Generated by NetworkManager
search myown.domain.com
nameserver 8.8.8.8
nameserver 8.8.8.4

Usage 4b: Using nmcli to update domain search (replace manual scripting of /etc/resolv.conf)

# nmcli con mod ens33 ipv4.dns-search 'myown.domain.com'
# nmcli con up ens33

Usage 5a: Disable IPv6 

# nmcli con mod ens33 ipv6.method "disabled"
# nmcli con up ens33
.....
....
ipv6.method:                            disabled
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
....
.....

Display the IP settings of the device. If there is no inet6 entry is displayed, IPv6 is disabled on the device.

# ip address show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
altname enp2s1
inet 192.168.x.x/19 brd 192.168.x.x scope global noprefixroute ens33
    valid_lft forever preferred_lft forever

References:

by

Disabling ipv6 on Rocky Linux 8 with Ansible

If you wish to disable ipv6 on Rocky Linux 8, there is a wonderful writeup on the script found at https://github.com/juju4/ansible-ipv6/blob/main/tasks/ipv6-disable.yml which you may find useful. If you just need to disable it temporarily without disruption (assuming you have not been using ipv6 at all)

- name: Disable IPv6 with sysctl
  ansible.posix.sysctl:
    name: "{{ item }}"
    value: "1"
    state: "present"
    reload: "yes"
  with_items:
    - net.ipv6.conf.all.disable_ipv6
    - net.ipv6.conf.default.disable_ipv6
    - net.ipv6.conf.lo.disable_ipv6

If you can tolerate a bit of disruption, you may want to take a look at putting it at the network configuration and restarting it

- name: RedHat | disable ipv6 in sysconfig/network
  ansible.builtin.lineinfile:
    dest: /etc/sysconfig/network
    regexp: "^{{ item.regexp }}"
    line: "{{ item.line }}"
    mode: '0644'
    backup: true
    create: true
  with_items:
    - { regexp: 'NETWORKING_IPV6=.*', line: 'NETWORKING_IPV6=NO' }
    - { regexp: 'IPV6INIT=.*', line: 'IPV6INIT=no' }
  notify:
    - Restart network
    - Restart NetworkManager
  when: ansible_os_family == 'RedHat'
by

Basic Understanding RoCE and Infiniband

Prerequisites:

  1. RoCE required Compliant Ethernet. Currently, I am using Mellanox ConnectX-6 Cards
  2. RoCE required a Compliant Switch. I used Mellanox 100G Switch.

The Difference between Traditional Ethernet Communication and RoCE can be explained very clearly in the diagram taken by Huawei’s Basic Knowledge and Differences of RoCE, IB, and TCP Networks

Some Key Pointers on the difference between TCP/IP and RDMA

  1. The Traditional TCP/IP network communication uses the Kernel to send messages which have high data movement and data replication overhead.
  2. RDMA can bypass the kernel and access the memory directly which allows low-latency network communication.

There are 3 types of RDMA network technologies is so neatly presented in Basic Knowledge and Differences of RoCE, IB, and TCP Networks

References:

  1. Basic Knowledge and Differences of RoCE, IB, and TCP Networks
by

Understanding the Difference between QSFP, QSFP+, QSFP28

Sometimes I use these terms loosely. Here an article that explain the 3 fiber optic transceivers QSFP, QSFP+ and QSFP28

Taken from the article “Difference between QSFP, QSFP+, QSFP28“

Here are some main points

  1. The QSFP specification supports Ethernet, Fibre Channel, InfiniBand and SONET/SDH standards with different data rate options.
  2. QSFP transceivers support the network link over singlemode or multimode fiber patch cable.
  3. Common ones are 4x10G QSFP+, 4x28G QSFP28
  4. QSFP+ are designed to support 40G Ethernet, Serial Attached SCSI, QDR (40G) and FDR (56G) Infiniband, and other communication standards
  5. QSFP+ modules integrate 4 transmit and 4 receive channels plus sideband signals. Then QSFP+ modules can break out into 4x10G lanes. 
  6. QSFP28 is a hot-pluggable transceiver module designed for 100G data rate.
  7. QSFP28 integrates 4 transmit and 4 receiver channels. “28” means each lane carries up to 28G data rate.
  8. QSFP28 can do 4x25G breakout connection, 2x50G breakout, or 1x100G depending on the transceiver used.
  9. Usually QSFP28 modules can’t break out into 10G links. But it’s another case to insert a QSFP28 module into a QSFP+ port if switches support.
  10. QSFP+ and QSFP28 modules can support both short and long-haul transmission.