Sometimes, you may encountered errors like this during an intensive run.

How do you know the value of the hard limit set? There is a good article by RedHat that explained “How is the nproc hard limit is calculated”

According to the article,

The limit depends on the total memory available on the server, which is calculated at boot time by the kernel as explained below:

/*
* Resource limit IDs
*
* ( Compatibility detail: there are architectures that have
* a different rlimit ID order in the 5-9 range and want
* to keep that order for binary compatibility. The reasons
* are historic and all new rlimits are identical across all
* arches. If an arch has such special order for some rlimits
* then it defines them prior including asm-generic/resource.h. )
*/

#define RLIMIT_CPU 0 /* CPU time in sec */
#define RLIMIT_FSIZE 1 /* Maximum filesize */
#define RLIMIT_DATA 2 /* max data size */
#define RLIMIT_STACK 3 /* max stack size */
#define RLIMIT_CORE 4 /* max core file size */

#ifndef RLIMIT_RSS
# define RLIMIT_RSS 5 /* max resident set size */
#endif

#ifndef RLIMIT_NPROC
# define RLIMIT_NPROC 6 /* max number of processes */
#endif

#ifndef RLIMIT_NOFILE
# define RLIMIT_NOFILE 7 /* max number of open files */
#endif

#ifndef RLIMIT_MEMLOCK
# define RLIMIT_MEMLOCK 8 /* max locked-in-memory address space */
#endif

#ifndef RLIMIT_AS
# define RLIMIT_AS 9 /* address space limit */
#endif

#define RLIMIT_LOCKS 10 /* maximum file locks held */
#define RLIMIT_SIGPENDING 11 /* max number of pending signals */
#define RLIMIT_MSGQUEUE 12 /* maximum bytes in POSIX mqueues */
#define RLIMIT_NICE 13 /* max nice prio allowed to raise to
0-39 for nice level 19 .. -20 */
#define RLIMIT_RTPRIO 14 /* maximum realtime priority */
#define RLIMIT_RTTIME 15 /* timeout for RT tasks in us */
#define RLIM_NLIMITS 16
8<---------- 8< ---------------- 8< ---------------- 8< --------

According to the article, For nproc, the limit is calculated in the kernel before the first process is forked in kernel/fork.c called by start_kernel:

>> init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
>> init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

Below is the path to the function :

>> start_kernel
> fork_init(totalram_pages)
> if (max_threads < 20) max_threads = 20;
> init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
> init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

>>>> RLIMIT_NPROC = max_threads/2

- The value of these variables are:

-> max_threads = mempages / (8 * THREAD_SIZE / PAGE_SIZE);
mempages comes from the function argument : fork_init(totalram_pages);
-> #define THREAD_ORDER 2
-> #define THREAD_SIZE (PAGE_SIZE << THREAD_ORDER)
-> PAGE_SIZE = 4096 (but useless)

- mempages is assigned in dmesg during the boot process, for example:

>> Memory: 36 989 916k/38797312k available (5516k kernel code, 1049156k absent, 758240k reserved, 6912k data, 1332k init)
mempages = 36989916k / PAGE_SIZE = 36989916k / 4096 = 9 247 479

- As an example:

RLIMIT_NPROC = (mempages / (8 * THREAD_SIZE / PAGE_SIZE)) / 2
= (mempages / (8 * (PAGE_SIZE << THREAD_ORDER) / PAGE_SIZE )) /2
= ( 9247479 / (8 * (4096 * 4) / 4096 )) / 2
= ( 9247479 / (8 * 4 )) /2
RLIMIT_NPROC = 14 4491.859375

To look at the values of the hard limits using BASH, you can use the command

ulimit -hn
4096

To modify the limits, do proceed to /etc/security/limits.d/20-nproc.conf to change the number

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 4096
root soft nproc unlimited

References:

1. How is the nproc hard limit calculated?
2. How to set nproc (Hard and Soft) Values in CentOS / RHEL 5,6,7