mount.nfs: requested NFS version or transport protocol is not supported

If you have encountered issues like

mount.nfs: requested NFS version or transport protocol is not supported

OR

mount.nfs4: Protocol not supported

To resolve this

Mount with NFS version 3 (with 4 verbose flags)

% mount -vvvv -t nfs -o vers=3 nfs-server:/share /mnt/nfs

References:

  1. Error “mount.nfs: requested NFS version or transport protocol is not supported” when attempting to mount an NFS share on Red Hat Enterprise Linux 6

Using nfsstat to troubleshoot NFS performance issues

The write-up is taken from RedHat Using nfsstat and nfsiostat to troubleshoot NFS performance issues on Linux

NFS relies on the existing network infrastructure, any glitches on the network may affect the performance of the connection. One of the tools that can be used is nfsstat

% yum install nfs-utils

The nfsstat command

The nfsstat command displays statistical information about the NFS and Remote Procedure Call (RPC) interfaces to the kernel.

On Server Side,

% nfsstat -s
Server rpc stats:
calls badcalls badclnt badauth xdrcall
107310012 0 0 0 0

The most important field to check is the badcalls, which represents the total number of calls rejected by the RPC layer. When the badcalls is greater than 0, than the underlying network needs to be checked, as there might be latency.

 

On NFS Client Side,

% nfsstat -c
Client rpc stats:
calls retrans authrefrsh
23158 0 23172

Client nfs v3:
null getattr setattr lookup access readlink
0 0% 7237 31% 7 0% 1443 6% 7874 34% 11 0%
read write create mkdir symlink mknod
578 2% 4548 19% 585 2% 1 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 51 0%
fsstat fsinfo pathconf commit
25 0% 10 0% 5 0% 781 3%

The client is doing well as it has relatively few retransmission requests. If you are encountering excessive retransmissions, you may want to adjust data transfer buffer sizes, which are specified by the mount command options rsize and size.

 

Check for dropped packets

Check dropped packet by running the following command on both the server and the client:

% nfsstat -o net
Client packet stats:
packets udp tcp tcpconn
0 0 0 0

Yum History and Using Yum to Roll Back updates

Getting History of YUM actions

Point 1: List Yum actions list

(base) [root@hpc-node1 ~]# yum history list
Loaded plugins: fastestmirror, langpacks
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
42 | root <root> | 2020-06-30 11:36 | I, U | 3
41 | 12345 | 2020-06-04 16:43 | Install | 2
40 | 12345 | 2020-06-04 16:26 | I, O, U | 878 E<
39 | 12345 | 2020-03-20 13:54 | Install | 1 >
38 | 12345 | 2020-03-20 13:46 | Install | 1
37 | 12345 | 2020-03-20 13:45 | Install | 2
36 | 12345 | 2020-03-20 13:44 | Install | 1
35 | 12345 | 2020-03-20 13:43 | Install | 28
34 | 12345 | 2020-03-20 12:52 | Update | 4
33 | 12345 | 2020-03-20 12:51 | Install | 1
32 | 12345 | 2020-03-20 12:43 | Update | 1
31 | 12345 | 2020-03-20 11:10 | Install | 1
30 | 12345 | 2020-03-20 10:53 | Install | 2
29 | 12345 | 2020-02-20 14:54 | Install | 1
28 | 12345 | 2020-01-23 16:10 | I, U | 2
27 | 12345 | 2020-01-15 15:03 | Update | 15
26 | 12345 | 2020-01-15 15:03 | I, U | 18
25 | 12345 | 2020-01-15 15:03 | Update | 3
24 | 12345 | 2019-11-07 11:00 | Install | 1 <
23 | 12345 | 2019-09-23 12:47 | Install | 1 >

Point 2: Undo the Yum action.

(base) [root@hpc-node1 boot]# yum history undo 42
Loaded plugins: fastestmirror, langpacks
Undoing transaction 42, from Tue Jun 30 11:36:18 2020
.....
.....
Running transaction
  Installing : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                                                             1/4
  Erasing    : ntp-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                       2/4
  Erasing    : autogen-libopts-5.18-5.el7.x86_64                                                                                                                                                        3/4
  Cleanup    : ntpdate-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                   4/4
  Verifying  : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                                                                     1/4
  Verifying  : ntp-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                       2/4
  Verifying  : autogen-libopts-5.18-5.el7.x86_64                                                                                                                                                        3/4
  Verifying  : ntpdate-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                   4/4

Removed:
  autogen-libopts.x86_64 0:5.18-5.el7                              ntp.x86_64 0:4.2.6p5-29.el7.centos.2                              ntpdate.x86_64 0:4.2.6p5-29.el7.centos.2

Installed:
  ntpdate.x86_64 0:4.2.6p5-29.el7.centos

Complete!

Point 3 – Check on the history of a particular package

yum history list ntpdate
Loaded plugins: fastestmirror, langpacks
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    43 | root               | 2020-07-06 11:31 | D, E           |    3
    42 | root               | 2020-06-30 11:36 | I, U           |    3
    40 | root               | 2020-06-04 16:26 | I, O, U        |  878 EE
     2 | root               | 2019-07-09 09:31 | I, U           | 1029 EE

Getting Useful Information on CPU and Configuration

Point 1. lscpu

To install

yum install util-linux

lscpu – (Print out information about CPU and its configuration)

[user1@myheadnode1 ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
Stepping: 4
CPU MHz: 3200.000
BogoMIPS: 6400.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
Flags: fpu .................

Point 2: hwloc-ls

To install hwloc-ls

yum install hwloc

hwloc – (Prints out useful information about the NUMA locality of devices and general hardware locality information)

[user1@myheadnode1 ~]# hwloc-ls
Machine (544GB total)
NUMANode L#0 (P#0 256GB)
Package L#0 + L3 L#0 (25MB)
L2 L#0 (1024KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#16)
L2 L#1 (1024KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#1)
PU L#3 (P#17)
L2 L#2 (1024KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
PU L#4 (P#2)
PU L#5 (P#18)
L2 L#3 (1024KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
PU L#6 (P#3)
PU L#7 (P#19)
L2 L#4 (1024KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
PU L#8 (P#4)
PU L#9 (P#20)
L2 L#5 (1024KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
PU L#10 (P#5)
PU L#11 (P#21)
L2 L#6 (1024KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
PU L#12 (P#6)
PU L#13 (P#22)
L2 L#7 (1024KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
PU L#14 (P#7)
PU L#15 (P#23)
.....
.....
.....

Point 3 – Check whether the Boost is on for AMD

Print out if CPU boost is on or off

cat /sys/devices/system/cpu/cpufreq/boost
1

References:

  1. Tuning Guard for AMD EPYC (pdf)

Turning ksm and ksmtuned off

In this blog, I will write on how to turn off KSM and ksmtuned since I do not need these services and save some unnecessary swapping activities on the disk.

What is KSM?

According to RedHat Site (8.4. KERNEL SAME-PAGE MERGING (KSM)),
Kernel same-page Merging (KSM), used by the KVM hypervisor, allows KVM guests to share identical memory pages. These shared pages are usually common libraries or other identical, high-use data. KSM allows for greater guest density of identical or similar guest operating systems by avoiding memory duplication……

KSM is a Linux feature which uses this concept in reverse. KSM enables the kernel to examine two or more already running programs and compare their memory. If any memory regions or pages are identical, KSM reduces multiple identical memory pages to a single page……

8.4.4 Kernel same-page merging (KSM) has a performance overhead which may be too large for certain environments or host systems. KSM may also introduce side channels that could be potentially used to leak information across guests. If this is a concern, KSM can be disabled on per-guest basis.

Deactivating KSM

# systemctl stop ksmtuned
Stopping ksmtuned:                                         [  OK  ]
# systemctl stop ksm
Stopping ksm:                                              [  OK  ]

To permanently deactivate KSM with the systemctl commands

# systemctl disable ksm
# systemctl disable ksmtuned

When KSM is disabled, any memory pages that were shared prior to deactivating KSM are still shared. To delete all of the PageKSM in the system, use the following command:

# echo 2 >/sys/kernel/mm/ksm/run

After this is performed, the khugepaged daemon can rebuild transparent hugepages on the KVM guest physical memory. Using # echo 0 >/sys/kernel/mm/ksm/run stops KSM, but does not unshare all the previously created KSM pages (this is the same as the # systemctl stop ksmtuned command).

References:

  1. Redhat – 8.4. Kernel Same-Page Merging (KSM)

Checking the Limits an application is imposed during run

If you wish to look at a specific application limits during run, you can do the following

pgrep fortcom
12345

* I used for fortcom, but it could be any application you wish to take a look.

cat /proc/12345/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 4096 2190327 processes
Max open files 1024 4096 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 2190327 2190327 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

* You can take a look that there is no limits to Max locked Memory and Max file locks are unlimited.

Another way to calculate shared memory swapping

Using ipcs utlities to find out information on shared memory utilisation which can be useful for analysing the performance of the system. Let’s says you want to measure how much memory has been swapped.

ipcs -mu
------ Shared Memory Status --------
segments allocated 55
pages allocated 6655333
pages resident  5661034
pages swapped   947522
Swap performance: 0 attempts     0 successes

where
-m is “information about active shared memory segments”
-u is “Show status summary”

You would need PAGE Memory

getconf PAGESIZE
4096

To provide us with the information in MB

echo "$((947522*4096/1024/1024)) MB"
3701 MB

 

Using strace to detect df hanging issues on NFS

strace is a wonderful tool to trace system calls and signals

I was hanging issues whenever I do a “df” and I was curious which file system is calling issues

strace df
.....
.....
stat("/run/user/1304561586", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/17132623", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/17149581", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/1304565184", {st_mode=S_IFDIR|0700, st_size=60, ...}) = 0
stat("/scratch",

It is obvious that /scratch file hang immediately after being launched.