Finding Top Processes using Highest Memory and CPU Usage in Linux

Read this Article from Find Top Running Processes by Highest Memory and CPU Usage in Linux. This is a quick way to view processes that consumed the largest RAM and CPU

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
   PID   PPID CMD                         %MEM %CPU
414699 414695 /usr/local/ansys_inc/v201/f 20.4 98.8
 30371      1 /usr/local/pbsworks/pbs_acc  0.2  1.0
 32241      1 /usr/local/pbsworks/pbs_acc  0.2  4.0
 30222      1 /usr/local/pbsworks/pbs_acc  0.2  0.6
  7191      1 /usr/local/pbsworks/dm_exec  0.1  0.8
 30595      1 /usr/local/pbsworks/pbs_acc  0.1  3.1
 30013      1 /usr/local/pbsworks/pbs_acc  0.1  0.3
 29602  29599 nginx: worker process        0.1  0.2
 29601  29599 nginx: worker process        0.1  0.3

The -o is to specify the output format. The -e is to select all processes. In order to sort in descending format, it hsould be –sort=%mem


Getting Useful Information on CPU and Configuration

Point 1. lscpu

To install

yum install util-linux

lscpu – (Print out information about CPU and its configuration)

[user1@myheadnode1 ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
Stepping: 4
CPU MHz: 3200.000
BogoMIPS: 6400.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
Flags: fpu .................

Point 2: hwloc-ls

To install hwloc-ls

yum install hwloc

hwloc – (Prints out useful information about the NUMA locality of devices and general hardware locality information)

[user1@myheadnode1 ~]# hwloc-ls
Machine (544GB total)
NUMANode L#0 (P#0 256GB)
Package L#0 + L3 L#0 (25MB)
L2 L#0 (1024KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#16)
L2 L#1 (1024KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#1)
PU L#3 (P#17)
L2 L#2 (1024KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
PU L#4 (P#2)
PU L#5 (P#18)
L2 L#3 (1024KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
PU L#6 (P#3)
PU L#7 (P#19)
L2 L#4 (1024KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
PU L#8 (P#4)
PU L#9 (P#20)
L2 L#5 (1024KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
PU L#10 (P#5)
PU L#11 (P#21)
L2 L#6 (1024KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
PU L#12 (P#6)
PU L#13 (P#22)
L2 L#7 (1024KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
PU L#14 (P#7)
PU L#15 (P#23)

Point 3 – Check whether the Boost is on for AMD

Print out if CPU boost is on or off

cat /sys/devices/system/cpu/cpufreq/boost


  1. Tuning Guard for AMD EPYC (pdf)

Checking the Limits an application is imposed during run

If you wish to look at a specific application limits during run, you can do the following

pgrep fortcom

* I used for fortcom, but it could be any application you wish to take a look.

cat /proc/12345/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 4096 2190327 processes
Max open files 1024 4096 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 2190327 2190327 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

* You can take a look that there is no limits to Max locked Memory and Max file locks are unlimited.

Another way to calculate shared memory swapping

Using ipcs utlities to find out information on shared memory utilisation which can be useful for analysing the performance of the system. Let’s says you want to measure how much memory has been swapped.

ipcs -mu
------ Shared Memory Status --------
segments allocated 55
pages allocated 6655333
pages resident  5661034
pages swapped   947522
Swap performance: 0 attempts     0 successes

-m is “information about active shared memory segments”
-u is “Show status summary”

You would need PAGE Memory

getconf PAGESIZE

To provide us with the information in MB

echo "$((947522*4096/1024/1024)) MB"
3701 MB


Using strace to detect df hanging issues on NFS

strace is a wonderful tool to trace system calls and signals

I was hanging issues whenever I do a “df” and I was curious which file system is calling issues

strace df
stat("/run/user/1304561586", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/17132623", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/17149581", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/1304565184", {st_mode=S_IFDIR|0700, st_size=60, ...}) = 0

It is obvious that /scratch file hang immediately after being launched.

Using Find and Tar Together to Backup and Archive

Point 1: If you wish to find files in a single folder and tar them into gzip-compressed archive. You can use a one-liner to do it.

find -maxdepth 1 -name '*.sh' | tar czf script.tgz -T -

“-maxdepth” refers to the current depth of 1 or current directory

“-T -” causes tar to read its list from a file rather than the command line. The “-” means standard input and output.

You should have a file is script.tgz


Point 2: If you wish to find files in a single folder and tar them into bzip2-compressed archive.

find -maxdepth 1 -name '*.sh' | tar cjf script.tgz -T -

Checking Disk Usage within the subfolders

I like this command which I often use to look into the dish usages at the sub folder level to check for large usages

du -h -d 1
1.3M    ./Espresso-BEEF
65M     ./MATLAB
478M    ./Abaqus
10G     ./COMSOL
8.3M    ./Gaussian2
316K    ./scripts
4.9M    ./NB07
647M    ./pytorch-GAN
31M     ./Gaussian
12G     .


  • -h refers to human-readable
  • -d refers to depth level. By default, it is 0 which is the same as summarize

Using BASH to read Keyboard Input

Sometimes, you may want to write a simple BASH menu system to help non-Linux users. Here is a sample sample.

echo "

Welcome to GCluster User Help Menu System

Please select:
1. Display Current Logged Session(s)
2. Display Current Home Space Utilisation
3. Check Running Process(s)
4. Check my Project Funds
5. Check for ABAQUS Licenses
6. Check for MATLAB Distributed Compute Licenses
7. Check for ANSYS Licenses
8. List running jobs from Project Members
0. Quit

read -p "Enter selection [0-8] > "

if [[ $REPLY =~ ^[0-8]$ ]]; then
        if [[ $REPLY == 0 ]]; then
                echo "Program Terminated."
        if [[ $REPLY == 1 ]]; then
                echo "Logged session: "
                w |grep $USER
        if [[ $REPLY == 2 ]]; then
                echo "Home Space Utilisation ($USER):"
                echo "(Please be patient.... It will take a while)"
                du -sh $HOME
        if [[ $REPLY == 3 ]]; then
                echo "Check my Running Processes on Login Node:"
                top -u $USER
        if [[ $REPLY == 4 ]]; then
                module load ams
                read -p "Enter your Project ID: " project_id
                project_summary $project_id
	if [[ $REPLY == 5 ]]; then
	if [[ $REPLY == 6 ]]; then
	if [[ $REPLY == 7 ]]; then 
	if [[ $REPLY == 8 ]]; then
		read -p "Enter your Project ID: " project_id
	 	countjobs=`qselect -P $project_id |wc -l`
		if [[ $countjobs -gt 0 ]]; then
			qstat -ans `qselect -P $project_id`
			echo "No Running Projects"
        echo "Invalid entry." >&2
        exit 1

Using BASH command Find to locate files – Part 2

Using User-Defined Actions:

P1. Prompt the user before findings.

[u1@node1 Downloads]$ find -type f -name 'rrr' -ok ls -l '{}' ';'
< ls ... ./rrr > ? Y
-rw-r--r-- 1 u1 users 0 Nov 19 10:36 ./rrr


  • {} is the symbolic representation of the current path
  • ; is the required delimiter indicating the end of the command
  • -ok can replace -exec so that the user can be prompted before executing any commands.


Pre-defined Action

P1. Find the file and delete the files immediately

[u1@node1 Downloads]$ find -type f -name '*.mov' -delete

P2. Find and quit once the match has been made

[u1@node1 Downloads]$ find -type f -name '*.mov' -quit

P3. Perform the equivalent of ls -dil on the match file.

[u1@node1 Downloads]$ find -type f -name '*.mov' -ls
4554823854 70266 -rw-r--r--   1 u1   users 62574861 Sep  5  2018 ./


Using Operators

[u1@node1 Downloads]$ find \(-type -f -not -name '*.txt' \) -or \( -type -r -not -name '*.doc' \)
  • Parentheses have special meaning to the shell. Putting a backslash character in front of the parenthesis help to prevent shell from interpreting them.



Set the maximum number of levels that find will descend when performing tests and actions.

[u1@hpc-gekko1 Downloads]$ find -maxdepth 2 -type f -name '*' -ls
4554823854 70266 -rw-r--r--   1 u1   users 62574861 Sep  5  2018 ./