How to increase the number of threads created by the NFS daemon for CENTOS 7

Taken from How to increase the number of threads created by the NFS daemon in RHEL 4, 5, 6 and 7?

In case of a NFS server with a high load, it may be advisable to increase the number of the threads created during the nfsd server start up.

Edit the following line in /etc/nfs.conf

% vim /etc/nfs.conf
#[nfsd]
# debug=0
threads=64
# host=
# port=0
# grace-time=90
# lease-time=90
# udp=y
# tcp=y

Testing whether it works….

% cat /proc/net/rpc/nfsd

According to the RH, “The first number is the total number of NFS server threads started. The second number indicates whether at any time all of the threads were running at once. The remaining numbers are a thread count time histogram.”

th 64 0 2.610 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

mount.nfs: requested NFS version or transport protocol is not supported

If you have encountered issues like

mount.nfs: requested NFS version or transport protocol is not supported

OR

mount.nfs4: Protocol not supported

To resolve this

Mount with NFS version 3 (with 4 verbose flags)

% mount -vvvv -t nfs -o vers=3 nfs-server:/share /mnt/nfs

References:

  1. Error “mount.nfs: requested NFS version or transport protocol is not supported” when attempting to mount an NFS share on Red Hat Enterprise Linux 6

Yum History and Using Yum to Roll Back updates

Getting History of YUM actions

Point 1: List Yum actions list

(base) [root@hpc-node1 ~]# yum history list
Loaded plugins: fastestmirror, langpacks
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
42 | root <root> | 2020-06-30 11:36 | I, U | 3
41 | 12345 | 2020-06-04 16:43 | Install | 2
40 | 12345 | 2020-06-04 16:26 | I, O, U | 878 E<
39 | 12345 | 2020-03-20 13:54 | Install | 1 >
38 | 12345 | 2020-03-20 13:46 | Install | 1
37 | 12345 | 2020-03-20 13:45 | Install | 2
36 | 12345 | 2020-03-20 13:44 | Install | 1
35 | 12345 | 2020-03-20 13:43 | Install | 28
34 | 12345 | 2020-03-20 12:52 | Update | 4
33 | 12345 | 2020-03-20 12:51 | Install | 1
32 | 12345 | 2020-03-20 12:43 | Update | 1
31 | 12345 | 2020-03-20 11:10 | Install | 1
30 | 12345 | 2020-03-20 10:53 | Install | 2
29 | 12345 | 2020-02-20 14:54 | Install | 1
28 | 12345 | 2020-01-23 16:10 | I, U | 2
27 | 12345 | 2020-01-15 15:03 | Update | 15
26 | 12345 | 2020-01-15 15:03 | I, U | 18
25 | 12345 | 2020-01-15 15:03 | Update | 3
24 | 12345 | 2019-11-07 11:00 | Install | 1 <
23 | 12345 | 2019-09-23 12:47 | Install | 1 >

Point 2: Undo the Yum action.

(base) [root@hpc-node1 boot]# yum history undo 42
Loaded plugins: fastestmirror, langpacks
Undoing transaction 42, from Tue Jun 30 11:36:18 2020
.....
.....
Running transaction
  Installing : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                                                             1/4
  Erasing    : ntp-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                       2/4
  Erasing    : autogen-libopts-5.18-5.el7.x86_64                                                                                                                                                        3/4
  Cleanup    : ntpdate-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                   4/4
  Verifying  : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                                                                     1/4
  Verifying  : ntp-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                       2/4
  Verifying  : autogen-libopts-5.18-5.el7.x86_64                                                                                                                                                        3/4
  Verifying  : ntpdate-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                   4/4

Removed:
  autogen-libopts.x86_64 0:5.18-5.el7                              ntp.x86_64 0:4.2.6p5-29.el7.centos.2                              ntpdate.x86_64 0:4.2.6p5-29.el7.centos.2

Installed:
  ntpdate.x86_64 0:4.2.6p5-29.el7.centos

Complete!

Point 3 – Check on the history of a particular package

yum history list ntpdate
Loaded plugins: fastestmirror, langpacks
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    43 | root               | 2020-07-06 11:31 | D, E           |    3
    42 | root               | 2020-06-30 11:36 | I, U           |    3
    40 | root               | 2020-06-04 16:26 | I, O, U        |  878 EE
     2 | root               | 2019-07-09 09:31 | I, U           | 1029 EE

How is the nproc hard limit calculated and how do we change the value on CentOS 7

Sometimes, you may encountered errors like this during an intensive run.

How do you know the value of the hard limit set? There is a good article by RedHat that explained “How is the nproc hard limit is calculated

According to the article,

The limit depends on the total memory available on the server, which is calculated at boot time by the kernel as explained below:

/*
* Resource limit IDs
*
* ( Compatibility detail: there are architectures that have
* a different rlimit ID order in the 5-9 range and want
* to keep that order for binary compatibility. The reasons
* are historic and all new rlimits are identical across all
* arches. If an arch has such special order for some rlimits
* then it defines them prior including asm-generic/resource.h. )
*/

#define RLIMIT_CPU 0 /* CPU time in sec */
#define RLIMIT_FSIZE 1 /* Maximum filesize */
#define RLIMIT_DATA 2 /* max data size */
#define RLIMIT_STACK 3 /* max stack size */
#define RLIMIT_CORE 4 /* max core file size */

#ifndef RLIMIT_RSS
# define RLIMIT_RSS 5 /* max resident set size */
#endif

#ifndef RLIMIT_NPROC
# define RLIMIT_NPROC 6 /* max number of processes */
#endif

#ifndef RLIMIT_NOFILE
# define RLIMIT_NOFILE 7 /* max number of open files */
#endif

#ifndef RLIMIT_MEMLOCK
# define RLIMIT_MEMLOCK 8 /* max locked-in-memory address space */
#endif

#ifndef RLIMIT_AS
# define RLIMIT_AS 9 /* address space limit */
#endif

#define RLIMIT_LOCKS 10 /* maximum file locks held */
#define RLIMIT_SIGPENDING 11 /* max number of pending signals */
#define RLIMIT_MSGQUEUE 12 /* maximum bytes in POSIX mqueues */
#define RLIMIT_NICE 13 /* max nice prio allowed to raise to
0-39 for nice level 19 .. -20 */
#define RLIMIT_RTPRIO 14 /* maximum realtime priority */
#define RLIMIT_RTTIME 15 /* timeout for RT tasks in us */
#define RLIM_NLIMITS 16
8<---------- 8< ---------------- 8< ---------------- 8< --------

According to the article, For nproc, the limit is calculated in the kernel before the first process is forked in kernel/fork.c called by start_kernel:

>> init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
>> init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

Below is the path to the function :

>> start_kernel
> fork_init(totalram_pages)
> if (max_threads < 20) max_threads = 20;
> init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
> init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

>>>> RLIMIT_NPROC = max_threads/2

- The value of these variables are:

-> max_threads = mempages / (8 * THREAD_SIZE / PAGE_SIZE);
mempages comes from the function argument : fork_init(totalram_pages);
-> #define THREAD_ORDER 2
-> #define THREAD_SIZE (PAGE_SIZE << THREAD_ORDER)
-> PAGE_SIZE = 4096 (but useless)

- mempages is assigned in dmesg during the boot process, for example:

>> Memory: 36 989 916k/38797312k available (5516k kernel code, 1049156k absent, 758240k reserved, 6912k data, 1332k init)
mempages = 36989916k / PAGE_SIZE = 36989916k / 4096 = 9 247 479

- As an example:

RLIMIT_NPROC = (mempages / (8 * THREAD_SIZE / PAGE_SIZE)) / 2
= (mempages / (8 * (PAGE_SIZE << THREAD_ORDER) / PAGE_SIZE )) /2
= ( 9247479 / (8 * (4096 * 4) / 4096 )) / 2
= ( 9247479 / (8 * 4 )) /2
RLIMIT_NPROC = 14 4491.859375

To look at the values of the hard limits using BASH, you can use the command

ulimit -hn
4096

To modify the limits, do proceed to /etc/security/limits.d/20-nproc.conf to change the number

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 4096
root soft nproc unlimited

 

References:

  1. How is the nproc hard limit calculated?
  2. How to set nproc (Hard and Soft) Values in CentOS / RHEL 5,6,7

Fixing Dependencies Resolution on CentOS 7

Taken from RedHat Solutions- “Yum update or yum install fails with package conflict between 64 bit and 32 bit package architectures ?”

You can configure the yum client to update only a package of the exact architecture installed on the system.

[root@node1 R-3.6.2]# yum install libcurl-devel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* centos-sclo-rh: mirror.0x.sg
Resolving Dependencies
--> Running transaction check
---> Package libcurl-devel.x86_64 0:7.29.0-42.el7 will be installed
--> Processing Dependency: libcurl = 7.29.0-42.el7 for package: libcurl-devel-7.29.0-42.el7.x86_64
--> Finished Dependency Resolution
Error: Package: libcurl-devel-7.29.0-42.el7.x86_64 (centos74-dvd)
Requires: libcurl = 7.29.0-42.el7
Installed: libcurl-7.29.0-46.el7.x86_64 (installed)
libcurl = 7.29.0-46.el7
Available: libcurl-7.29.0-42.el7.x86_64 (centos74-dvd)
libcurl = 7.29.0-42.el7
You could try using --skip-broken to work around the problem
** Found 139 pre-existing rpmdb problem(s), 'yum check' output follows:

 

You can configure the yum client to update only a package of the exact architecture installed on the system.

1. Install the yum-utils package

yum install yum-utils

2. The package-cleanup –dupes lists all duplicate packages:

# package-cleanup --dupes

3. The package-cleanup –cleandupes removes the duplicates

# package-cleanup --cleandupes

4. Edit /etc/yum.conf, set the following line for CentOS 7:

exactarchlist=*

Run yum command:

# yum clean all
# yum update

References:

  1. “Yum update or yum install fails with package conflict between 64 bit and 32 bit package architectures ?”

Using firewall-cmd rich rules to whitelist IP Address Range

For basic firewall-cmd Using firewall-cmd in CentOS 7

For starting and stopping firewalld service Disable FirewallD Services on CentOS 7

Firewall Rich Rules are additional feature of firewalld that allows you to create most sophisticated firewall rules.

Option 1a: To add a rich rule to allow a subnet to be whitelist

# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="22" protocol="tcp" accept'

Option 1b: To add a rule rule to allow a service to be whitelist

# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" service name="ssh" accept'

 

Option 1c: To remove a rich rule to allow a subnet to be whitelist

# firewall-cmd --permanent --zone=public --remove-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="22" protocol="tcp" accept'

Option 2a: To add log entry

# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24 port port="22" protocol="tcp" log prefix="Firewall Rich Rule Log" level="notice" accept'

Option 3a: Port Forwarding

# firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24 port port="22" protocol="tcp" log prefix="Firewall Rich Rule Log " level="notice" forward-port port="11111" to-port="22" accept'

Option 3b: Testing

# ssh -p 11111 admin@myip.com

Redirecting to another site in User Directory in APACHE

Sometimes your users may require you to redirect to their new site and even capture errors like 404, 401 in their old site and redirect to their new site. You can do it by doing the following.

Step 1: After clearing out all the old site, you may want to put in a redirection page. Can be a simple one line on a index.html

Step 2: To trap “missing or ghost directory and files” in the website of the user directory. For example the old site could be http://www.myoldsite.com/~me/

You may want to create a .conf file such as myoldsite.conf and placed it at /etc/httpd/conf.d and put in the following configuration.

Step 3: Update the httpd service.

For CentOS 7, it could be

# systemctl start httpd.service

Remove virbr0 Interfaces from CentOS 7

Step 1: Stop the libvirtd Service

# systemctl stop libvirtd.service

You should see something like this

● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2019-06-03 13:33:26 +08; 43s ago
Docs: man:libvirtd(8)
https://libvirt.org
Process: 28069 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 28069 (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 32768)
Memory: 6.7M
CGroup: /system.slice/libvirtd.service
├─28896 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
└─28897 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

Step 2. Disable the service

# systemctl disable libvirtd.service

Step 3: Removing the virbr0 interfaces on machine

# brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242f3432864       no
virbr0          8000.525400d6fcaa       yes             virbr0-nic

Step 4: Down the Bridge Link

# ip link set virbr0 down

Step 5: Remove the Bridge

# brctl delbr virbr0

Step 6: Verify that the bridge has been removed.

# brtcl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242f3432864 no

rpcbind.socket systemd unit fails to start when IPv6 is disabled

I encountered this error when I used this command

echo “net.ipv6.conf.all.disable_ipv6 = 1” >> /etc/sysctl.d/ipv6.conf

When I rebooted the server, my NFS Services were dysfunctional. The rpcbind.socket systemd unit fails to load. I managed to find information on Red Hat Bugzilla – Bug 1402961 rpcbind.socket systemd unit fails to start when IPv6 is disabled. 

The Solution is simply remove echo “net.ipv6.conf.all.disable_ipv6 = 0” >> /etc/sysctl.d/ipv6.conf