Error “Too many files open” on CentOS 7

If you are encountering Error messages during login with “Too many open files” and the session gets terminated automatically, it is because the open file limit for a user or system exceeds the default setting and  you may wish to change it

@ System Levels

To see the settings for maximum open files,

# cat /proc/sys/fs/file-max
55494980

This value means that the maximum number of files all processes running on the system can open. By default this number will automatically vary according to the amount of RAM in the system. As a rough guideline it will be about 100,000 files per GB of RAM.

 

To override the system wide maximum open files, as edit the /etc/sysctl.conf

# vim /etc/sysctl.conf
 fs.file-max = 80000000

Activate this change to the live system

# sysctl -p

@ User Level

To see the setting for maximum open files for a user

# su - user1
$ ulimit -n
1024

To change the setting, edit the /etc/security/limits.conf

$ vim /etc/security/limits.conf
user - nofile 2048

To change for all users

* - nofile 2048

This set the maximum open files for ALL users to 2048 files. These settings will require a reboot to activate.

References:

  1. How to correct the error “Too many files open” on Red Hat Enterprise Linux

Tools to Show your System Configuration

Tool 1: Display Information about CPU Architecture

[user1@node1 ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
Stepping: 4
CPU MHz: 3200.000
BogoMIPS: 6400.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
.....
.....

Tool 2: List all PCI devices

[user1@node1 ~]# lspci -t -vv
-+-[0000:d7]-+-00.0-[d8]--
| +-01.0-[d9]--
| +-02.0-[da]--
| +-03.0-[db]--
| +-05.0 Intel Corporation Device 2034
| +-05.2 Intel Corporation Sky Lake-E RAS Configuration Registers
| +-05.4 Intel Corporation Device 2036
| +-0e.0 Intel Corporation Device 2058
| +-0e.1 Intel Corporation Device 2059
| +-0f.0 Intel Corporation Device 2058
| +-0f.1 Intel Corporation Device 2059
| +-10.0 Intel Corporation Device 2058
| +-10.1 Intel Corporation Device 2059
| +-12.0 Intel Corporation Sky Lake-E M3KTI Registers
| +-12.1 Intel Corporation Sky Lake-E M3KTI Registers
| +-12.2 Intel Corporation Sky Lake-E M3KTI Registers
| +-12.4 Intel Corporation Sky Lake-E M3KTI Registers
| +-12.5 Intel Corporation Sky Lake-E M3KTI Registers
| +-15.0 Intel Corporation Sky Lake-E M2PCI Registers
| +-16.0 Intel Corporation Sky Lake-E M2PCI Registers
| +-16.4 Intel Corporation Sky Lake-E M2PCI Registers
| \-17.0 Intel Corporation Sky Lake-E M2PCI Registers
.....
.....

Tool 3: List all PCI devices

[user@node1 ~]# lsblk
sda 8:0 0 1.1T 0 disk
├─sda1 8:1 0 200M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 1.1T 0 part
├─centos-root 253:0 0 50G 0 lvm /
├─centos-swap 253:1 0 4G 0 lvm [SWAP]
└─centos-home 253:2 0 1T 0 lvm

Tool 4: See flags kernel booted with

[user@node1 ~]
BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8

Tool 5: Display available network interfaces

[root@hpc-gekko1 ~]# ifconfig -a
eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether XX:XX:XX:XX:XX:XX txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
.....
.....
.....

Tool 6 Using dmidecode to find hardware information
See Using dmidecode to find hardware information

How to configure NFS on CentOS 7

Step 1: Do a Yum Install

# yum install nfs-utils rpcbind

Step 2: Enable the Service at Boot Time

# systemctl enable nfs-server
# systemctl enable rpcbind
# systemctl enable nfs-lock     (it does not need to be enabled since rpc-statd.service  is static.)
# systemctl enable nfs-idmap    (it does not need to be enabled since nfs-idmapd.service is static.)

Step 3: Start the Services

# systemctl start rpcbind
# systemctl start nfs-server
# systemctl start nfs-lock
# systemctl start nfs-idmap

Step 4: Confirm the status of NFS

# systemctl status nfs

Step 5: Create a mount point

# mkdir /shared-data

Step 6: Exports the Share

# vim /etc/exports
/shared-data 192.168.0.0/16(rw,no_root_squash)

Step 7: Export the Share

# exportfs -rv

Step 8: Restart the NFS Services

# systemctl restart nfs-server

Step 9: Configure the Firewall

# firewall-cmd --add-service=nfs --zone=internal --permanent
# firewall-cmd --add-service=mountd --zone=internal --permanent
# firewall-cmd --add-service=rpc-bind --zone=internal --permanent

References:

  1. How to configure NFS in RHEL 7
  2. What firewalld services should be active on an NFS server in RHEL 7?

Configuring External libraries for R-Studio for CentOS 7

RStudio can be configured by adding entries to 2 configuration files. It may not exist by default and you may need to create

/etc/rstudio/rserver.conf
/etc/rstudio/rsession.conf

If you need to add additional libraries to the default LD_LIBRARY_PATH for R sessions. You have to add the parameters to the /etc/rstudio/rserver.conf

Step 1: Enter External Libraries settings
For example, you may want to add GCC-6.5 libraries

# Server Configuration File
rsession-ld-library-path=/usr/local/gcc-6.5.0/lib64:/usr/local/gcc-6.5.0/lib

Step 2a: Stop the R Server Services

# /usr/sbin/rstudio-server stop

Step 2b: Verify the Installation

# rstudio-server verify-installation

Step 2c: Start the R Server Services

# /usr/sbin/rstudio-server start
# /usr/sbin/rstudio-server status

(Make sure there is no error)

References:

  1. RStudio Server: Configuring the Server

 

Installing R-Studio on CentOS-7

Prerequisites

  • R-Studio (rstudio-server-rhel-1.3.1093-x86_64.rpm)
  • R-3.6.3
  • GNU-6.5
  • m4-1.4.18
  • gmp-6.1.0
  • mpfr-3.1.4
  • mpc-1.0.3
  • isl-0.18
  • gsl-2.1

Step 1: Download R-Studio v1.3.1093

On your terminal, Download the free version RStudio Server v1.3.1093 from RStudio.com

Step 2: Install R-Studio v1.3.1093

Follow the Install Process as written in https://rstudio.com/products/rstudio/download-server/redhat-centos/

% wget https://download2.rstudio.org/server/centos6/x86_64/rstudio-server-rhel-1.3.1093-x86_64.rpm
% yum install rstudio-server-rhel-1.3.1093-x86_64.rpm

Step 3: Verify Installation

% /usr/sbin/rstudio-server verify-installation

Step 4: Make sure the R is compiled and placed correctly

  • Make sure the R –enable-R-shlib flags
  • RStudio will search for your installation of R in the traditional locations, such as RStudio will search for your installation of R in the traditional locations, such as
    /usr/bin/R
    /usr/local/bin/R

    If R is not installed in these locations, RStudio may not be able to find it.

Step 5: Fixing Login Issues

If you managed to launch the R-Studio,

 

But not able to or having error. Do take a look at PAM authentication in RStudio Connect

Simply copy /etc/pam.d/login and replace the /etc/pam.d/rstudio

% cp /etc/pam.d/login /etc/pam.d/rstudio

 

Useful References:

  1. RStudio Server Will Not Start
  2. PAM authentication in RStudio Connect

How to increase the number of threads created by the NFS daemon for CENTOS 7

Taken from How to increase the number of threads created by the NFS daemon in RHEL 4, 5, 6 and 7?

In case of a NFS server with a high load, it may be advisable to increase the number of the threads created during the nfsd server start up.

Edit the following line in /etc/nfs.conf

% vim /etc/nfs.conf
#[nfsd]
# debug=0
threads=64
# host=
# port=0
# grace-time=90
# lease-time=90
# udp=y
# tcp=y

Testing whether it works….

% cat /proc/net/rpc/nfsd

According to the RH, “The first number is the total number of NFS server threads started. The second number indicates whether at any time all of the threads were running at once. The remaining numbers are a thread count time histogram.”

th 64 0 2.610 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

mount.nfs: requested NFS version or transport protocol is not supported

If you have encountered issues like

mount.nfs: requested NFS version or transport protocol is not supported

OR

mount.nfs4: Protocol not supported

To resolve this

Mount with NFS version 3 (with 4 verbose flags)

% mount -vvvv -t nfs -o vers=3 nfs-server:/share /mnt/nfs

References:

  1. Error “mount.nfs: requested NFS version or transport protocol is not supported” when attempting to mount an NFS share on Red Hat Enterprise Linux 6

Yum History and Using Yum to Roll Back updates

Getting History of YUM actions

Point 1: List Yum actions list

(base) [root@hpc-node1 ~]# yum history list
Loaded plugins: fastestmirror, langpacks
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
42 | root <root> | 2020-06-30 11:36 | I, U | 3
41 | 12345 | 2020-06-04 16:43 | Install | 2
40 | 12345 | 2020-06-04 16:26 | I, O, U | 878 E<
39 | 12345 | 2020-03-20 13:54 | Install | 1 >
38 | 12345 | 2020-03-20 13:46 | Install | 1
37 | 12345 | 2020-03-20 13:45 | Install | 2
36 | 12345 | 2020-03-20 13:44 | Install | 1
35 | 12345 | 2020-03-20 13:43 | Install | 28
34 | 12345 | 2020-03-20 12:52 | Update | 4
33 | 12345 | 2020-03-20 12:51 | Install | 1
32 | 12345 | 2020-03-20 12:43 | Update | 1
31 | 12345 | 2020-03-20 11:10 | Install | 1
30 | 12345 | 2020-03-20 10:53 | Install | 2
29 | 12345 | 2020-02-20 14:54 | Install | 1
28 | 12345 | 2020-01-23 16:10 | I, U | 2
27 | 12345 | 2020-01-15 15:03 | Update | 15
26 | 12345 | 2020-01-15 15:03 | I, U | 18
25 | 12345 | 2020-01-15 15:03 | Update | 3
24 | 12345 | 2019-11-07 11:00 | Install | 1 <
23 | 12345 | 2019-09-23 12:47 | Install | 1 >

Point 2: Undo the Yum action.

(base) [root@hpc-node1 boot]# yum history undo 42
Loaded plugins: fastestmirror, langpacks
Undoing transaction 42, from Tue Jun 30 11:36:18 2020
.....
.....
Running transaction
  Installing : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                                                             1/4
  Erasing    : ntp-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                       2/4
  Erasing    : autogen-libopts-5.18-5.el7.x86_64                                                                                                                                                        3/4
  Cleanup    : ntpdate-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                   4/4
  Verifying  : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                                                                     1/4
  Verifying  : ntp-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                       2/4
  Verifying  : autogen-libopts-5.18-5.el7.x86_64                                                                                                                                                        3/4
  Verifying  : ntpdate-4.2.6p5-29.el7.centos.2.x86_64                                                                                                                                                   4/4

Removed:
  autogen-libopts.x86_64 0:5.18-5.el7                              ntp.x86_64 0:4.2.6p5-29.el7.centos.2                              ntpdate.x86_64 0:4.2.6p5-29.el7.centos.2

Installed:
  ntpdate.x86_64 0:4.2.6p5-29.el7.centos

Complete!

Point 3 – Check on the history of a particular package

yum history list ntpdate
Loaded plugins: fastestmirror, langpacks
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    43 | root               | 2020-07-06 11:31 | D, E           |    3
    42 | root               | 2020-06-30 11:36 | I, U           |    3
    40 | root               | 2020-06-04 16:26 | I, O, U        |  878 EE
     2 | root               | 2019-07-09 09:31 | I, U           | 1029 EE

How is the nproc hard limit calculated and how do we change the value on CentOS 7

Sometimes, you may encountered errors like this during an intensive run.

How do you know the value of the hard limit set? There is a good article by RedHat that explained “How is the nproc hard limit is calculated

According to the article,

The limit depends on the total memory available on the server, which is calculated at boot time by the kernel as explained below:

/*
* Resource limit IDs
*
* ( Compatibility detail: there are architectures that have
* a different rlimit ID order in the 5-9 range and want
* to keep that order for binary compatibility. The reasons
* are historic and all new rlimits are identical across all
* arches. If an arch has such special order for some rlimits
* then it defines them prior including asm-generic/resource.h. )
*/

#define RLIMIT_CPU 0 /* CPU time in sec */
#define RLIMIT_FSIZE 1 /* Maximum filesize */
#define RLIMIT_DATA 2 /* max data size */
#define RLIMIT_STACK 3 /* max stack size */
#define RLIMIT_CORE 4 /* max core file size */

#ifndef RLIMIT_RSS
# define RLIMIT_RSS 5 /* max resident set size */
#endif

#ifndef RLIMIT_NPROC
# define RLIMIT_NPROC 6 /* max number of processes */
#endif

#ifndef RLIMIT_NOFILE
# define RLIMIT_NOFILE 7 /* max number of open files */
#endif

#ifndef RLIMIT_MEMLOCK
# define RLIMIT_MEMLOCK 8 /* max locked-in-memory address space */
#endif

#ifndef RLIMIT_AS
# define RLIMIT_AS 9 /* address space limit */
#endif

#define RLIMIT_LOCKS 10 /* maximum file locks held */
#define RLIMIT_SIGPENDING 11 /* max number of pending signals */
#define RLIMIT_MSGQUEUE 12 /* maximum bytes in POSIX mqueues */
#define RLIMIT_NICE 13 /* max nice prio allowed to raise to
0-39 for nice level 19 .. -20 */
#define RLIMIT_RTPRIO 14 /* maximum realtime priority */
#define RLIMIT_RTTIME 15 /* timeout for RT tasks in us */
#define RLIM_NLIMITS 16
8<---------- 8< ---------------- 8< ---------------- 8< --------

According to the article, For nproc, the limit is calculated in the kernel before the first process is forked in kernel/fork.c called by start_kernel:

>> init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
>> init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

Below is the path to the function :

>> start_kernel
> fork_init(totalram_pages)
> if (max_threads < 20) max_threads = 20;
> init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
> init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

>>>> RLIMIT_NPROC = max_threads/2

- The value of these variables are:

-> max_threads = mempages / (8 * THREAD_SIZE / PAGE_SIZE);
mempages comes from the function argument : fork_init(totalram_pages);
-> #define THREAD_ORDER 2
-> #define THREAD_SIZE (PAGE_SIZE << THREAD_ORDER)
-> PAGE_SIZE = 4096 (but useless)

- mempages is assigned in dmesg during the boot process, for example:

>> Memory: 36 989 916k/38797312k available (5516k kernel code, 1049156k absent, 758240k reserved, 6912k data, 1332k init)
mempages = 36989916k / PAGE_SIZE = 36989916k / 4096 = 9 247 479

- As an example:

RLIMIT_NPROC = (mempages / (8 * THREAD_SIZE / PAGE_SIZE)) / 2
= (mempages / (8 * (PAGE_SIZE << THREAD_ORDER) / PAGE_SIZE )) /2
= ( 9247479 / (8 * (4096 * 4) / 4096 )) / 2
= ( 9247479 / (8 * 4 )) /2
RLIMIT_NPROC = 14 4491.859375

To look at the values of the hard limits using BASH, you can use the command

ulimit -hn
4096

To modify the limits, do proceed to /etc/security/limits.d/20-nproc.conf to change the number

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 4096
root soft nproc unlimited

 

References:

  1. How is the nproc hard limit calculated?
  2. How to set nproc (Hard and Soft) Values in CentOS / RHEL 5,6,7

Fixing Dependencies Resolution on CentOS 7

Taken from RedHat Solutions- “Yum update or yum install fails with package conflict between 64 bit and 32 bit package architectures ?”

You can configure the yum client to update only a package of the exact architecture installed on the system.

[root@node1 R-3.6.2]# yum install libcurl-devel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* centos-sclo-rh: mirror.0x.sg
Resolving Dependencies
--> Running transaction check
---> Package libcurl-devel.x86_64 0:7.29.0-42.el7 will be installed
--> Processing Dependency: libcurl = 7.29.0-42.el7 for package: libcurl-devel-7.29.0-42.el7.x86_64
--> Finished Dependency Resolution
Error: Package: libcurl-devel-7.29.0-42.el7.x86_64 (centos74-dvd)
Requires: libcurl = 7.29.0-42.el7
Installed: libcurl-7.29.0-46.el7.x86_64 (installed)
libcurl = 7.29.0-46.el7
Available: libcurl-7.29.0-42.el7.x86_64 (centos74-dvd)
libcurl = 7.29.0-42.el7
You could try using --skip-broken to work around the problem
** Found 139 pre-existing rpmdb problem(s), 'yum check' output follows:

 

You can configure the yum client to update only a package of the exact architecture installed on the system.

1. Install the yum-utils package

yum install yum-utils

2. The package-cleanup –dupes lists all duplicate packages:

# package-cleanup --dupes

3. The package-cleanup –cleandupes removes the duplicates

# package-cleanup --cleandupes

4. Edit /etc/yum.conf, set the following line for CentOS 7:

exactarchlist=*

Run yum command:

# yum clean all
# yum update

References:

  1. “Yum update or yum install fails with package conflict between 64 bit and 32 bit package architectures ?”