One of our Linux Compute Server was showing when a particular was attempting to login on.
failed to execute /bin/bash: resource temporarily unavailable
We suspected that nprocs limits have been breached by the particular user. I found this write-up https://blog.dbi-services.com/linux-how-to-monitor-the-nproc-limit-1/ very prescriptive of the issue I faced.
Extracting information via ps is not useful unless you use the “-L” to show threads, possibly LWP (light-weight process).
% ps h -LA -o user | sort | uniq -c | sort -n 1 chrony 1 dbus 1 libstoragemgmt 1 nobody 1 rpc 1 rpcuser 2 avahi 2 user3 2 postfix 3 colord 3 rtkit 4 user1 4 user2 7 polkitd 23 user4 31 user5 34 user6 361 user7 442 user8 556 gdm 563 user9 922 user10 16384 user11 3319 root
You can see that user11 has 16384 threads!
To dig down into what is happening to a selected user. We will use user2 since it has one of the fewest LWP to
% ps -o nlwp,pid,lwp,args -u user2 | sort -n NLWP PID LWP COMMAND 1 272705 272705 sshd: user2@pts/12 1 273054 273054 sshd: user2@notty 1 273216 273216 /usr/libexec/openssh/sftp-server 1 273406 273406 -bash
nlwp – Number of LWP
lwp – Process of ID of the LWP.
To eliminate the offending user11’s thousands of threads
% pkill -KILL -u user11
References
The Linux Cluster 