BASH

by

Sorting ls Output Based on Time and Date

You may want to list the files in a directory based on last modified time and date from the latest to the oldest, you will need to use -t 

% ls -lt
drwxr-xr-x  26 root root    4096 Mar 29 13:40 mono
drwxr-xr-x   2 root root      42 Mar 29 13:40 mono-source-libs
lrwxrwxrwx   1 root root      22 Dec 21 12:55 libnss_centrifyda.so.0 -> libnss_centrifyda.so.2
drwxr-xr-x.  2 root root      62 Dec 21 12:54 security
lrwxrwxrwx   1 root root      22 Dec 21 12:54 libnss_centrifydc.so.0 -> libnss_centrifydc.so.2
lrwxrwxrwx   1 root root      28 Dec 21 12:54 libnss_centrify_logonly.so.0 -> libnss_centrify_logonly.so.2
-rwxr-xr-x   1 root root  238560 Nov 17 09:13 libnss_centrifydc.so.2
-rwxr-xr-x   1 root root  140064 Nov 17 09:13 libnss_centrify_logonly.so.2
-rwxr-xr-x   1 root root  159112 Nov  9 02:39 libnss_centrifyda.so.2
drwxr-xr-x   4 root root      40 Jul 25  2021 fastx
drwxr-xr-x.  2 root root     100 Dec 11  2020 modprobe.d
drwxr-xr-x   8 root root     180 Oct 26  2020 rstudio-server
.....
.....

If you want a sorting from the oldest to the latest, you use the -r option

% ls -ltr
.....
.....
-rwxr-xr-x   1 root root  159112 Nov  9 02:39 libnss_centrifyda.so.2
-rwxr-xr-x   1 root root  140064 Nov 17 09:13 libnss_centrify_logonly.so.2
-rwxr-xr-x   1 root root  238560 Nov 17 09:13 libnss_centrifydc.so.2
lrwxrwxrwx   1 root root      28 Dec 21 12:54 libnss_centrify_logonly.so.0 -> libnss_centrify_logonly.so.2
lrwxrwxrwx   1 root root      22 Dec 21 12:54 libnss_centrifydc.so.0 -> libnss_centrifydc.so.2
drwxr-xr-x.  2 root root      62 Dec 21 12:54 security
lrwxrwxrwx   1 root root      22 Dec 21 12:55 libnss_centrifyda.so.0 -> libnss_centrifyda.so.2
drwxr-xr-x   2 root root      42 Mar 29 13:40 mono-source-libs
drwxr-xr-x  26 root root    4096 Mar 29 13:40 mono
by

Listing 50 biggest files in Recursive Directories.

I have always enjoyed BASH Commands that can reveal more the files and directories we have. I wrote a short write-up Checking Disk Usage within the subfolders but avoid mount-point

I wondered whether if I can capture the top 20 directories that uses the most disk space. There was this document by OpenSource.com that I find helpful 7 Linux command-line tips for saving media file space

1 To Find the 50 biggest files in its recursive directory tree

% find  -type f  -exec  du -Sh {} +  |  sort -rh  |  head -n 50
19G     ./Downloads/xxxxxx.iso
17G     ./Downloads/AI/AI.tar
17G     ./Downloads/ISO/xxxx.iso
6.4G    ./Ansys/uuuuuuuu.zip
5.3G    ./.cache/tracker/meta.db
4.5G    ./Downloads/HelloThere_AVX2/tar/E45.tbJ
4.4G    ./Documents/EEEEE/EEEEEE_dvd.iso
3.7G    ./Downloads/12345/2021/2021.tar
.....
......
by

Using /proc/$PID/fd to understand where the files are written to.

/proc/$PID/fd provide a symbolic link to the file that the process $PID has open on the file descriptor NUM. One useful gain from this, is that I can use /proc/$PID/fd to understand where the files are written to. 

% top

Let’s use Process 66090 (1508.exe)

% cd /proc/66090/fd
% ls -l
lr-x------ 1 user1 user1 64 Feb 15 12:41 0 -> /localhome/user1/scratch/G16c-Mistral909223.hpc-mn1/G16c-Mistral.Input.909223.hpc-mn1
l-wx------ 1 user1 user1 64 Feb 15 12:41 1 -> /localhome/user1/AgMPYH-GSH_4.gjf.Output.909223.hpc-mn1.log
lrwx------ 1 user1 user1 64 Feb 15 12:42 10 -> /localhome/user1/scratch/G16c-Mistral909223.hpc-mn1/Gau-66090.d2e
l-wx------ 1 user1 user1 64 Feb 15 12:41 129 -> /myhome/user1/breezetraces/user1-909223.hpc-mn1-hpc-g8004/tmp/monitor-command-fifo
lr-x------ 1 user1 user1 64 Feb 15 12:41 130 -> /proc/66090/mounts
lrwx------ 1 user1 user1 64 Feb 15 12:41 2 -> /var/spool/pbs/spool/909223.hpc-mn1.ER
lrwx------ 1 user1 user1 64 Feb 15 12:41 3 -> /localhome/user1/scratch/G16c-Mistral909223.hpc-mn1/Gau-66081.inp
lrwx------ 1 user1 user1 64 Feb 15 12:41 4 -> socket:[75030280]
lrwx------ 1 user1 user1 64 Feb 15 12:41 5 -> socket:[75030281]
lrwx------ 1 user1 user1 64 Feb 15 12:42 6 -> /localhome/user1/scratch/G16c-Mistral909223.hpc-mn1/Gau-66090.rwf
lrwx------ 1 user1 user1 64 Feb 15 12:42 7 -> /localhome/user1/AgMPYH-GSH_4.chk
lrwx------ 1 user1 user1 64 Feb 15 12:42 8 -> /localhome/user1/scratch/G16c-Mistral909223.hpc-mn1/Gau-66090.skr
lrwx------ 1 user1 user1 64 Feb 15 12:42 9 -> /localhome/user1/scratch/G16c-Mistral909223.hpc-mn1/Gau-66090.int

You can see for most of the user processes, it is written to /localhome/user1.

by

/bin/rm : Argument list too long

I was trying to delete the files in the folder that has more than 90,000. When I delete the files, I had errors

% /bin/rm : Argument list too long

The issue is that there are too many files and rm is not able to clear. To workaround the issues, you can do the following. Make sure you are in the directory where you want to clear the file. 

 % find . -name '*' | xargs rmfind . -name '*' | xargs rm

References:

by

Checking Disk Usage within the subfolders but avoid mount-point

If you need to check Usage, but you wish to avoid the mount-point, you can use the command

[root@hpc-hn /]# du -h -x -d 1
48M     ./etc
552M    ./root
11G     ./var
1.1G    ./tmp
11G     ./usr
0       ./media
0       ./mnt
4.8G    ./opt
0       ./srv
0       ./install
0       ./log
0       ./misc
0       ./net
0       ./server_priv
0       ./ProjectSpace
0       ./media1
0       ./media2
28G     .
  • -h refers to human-readable
  • -d refers to depth level. By default, it is 0 which is the same as summarize
  • -x skip directories on different file systems
by

Basic CURL Commands

curl is a command line tool to transfer data to or from a server. It is able to use any of the supported protocols like HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP or FILE. This tool is very useful for automation, since it is designed to work without user interaction. Furthermore, curl can transfer multiple file at once.

Basic Single URL Usage

% curl https://thelinuxcluster.com/

2a. Save the Download File with a preferred file name

Save the Download File on the local machine with the name provided with the parameter.

% curl -o test.o https://thelinuxcluster.com/test.output

2b. Save the Download File

% curl -O https://thelinuxcluster.com/test.output

2c. Download Multiple Files. Just Multiple -O

% curl -O https://thelinuxcluster.com/CentOS1.iso -O https://thelinuxcluster.com/CentOS2.iso -O https://thelinuxcluster.com/CentOS3.iso

3a. Display a Progress Meter

% curl --progress-bar -o test.o https://thelinuxcluster.com/test.output

3b. Do not display a Progressive Bar

% curl --silent -o test.o https://thelinuxcluster.com/test.output

4 Limit Rate of Data Transfer

% curl --limit-rate 1000K -o test.o https://thelinuxcluster.com/CentOS1.iso

5a Uploading a File to the FTP Server

% curl -u username:userpassword -T myfile ftp://ftp.thelinuxcluster.com/

5b. Appending the File to the FTP Server

% curl -u username:userpassword -a -T myfile ftp://ftp.thelinuxcluster.com/

5c Downloading the File to the File Server

% curl ftp:/ftp.thelinuxcluster.com/CentOS79.iso --user username:userpassword -o myCentOS79.iso

6a. Verifying SSL Certificate

% curl --cacert server.crt https://thesupersecureserver.com

6b. Ignoring SSL Certificate

% curl -k https://thesupersecureserver.com/

7a Proxy Server

% curl -x proxy_name:proxy_port https://thelinuxcluster.com

7b Proxy Server which requires authentication

% curl --user username:userpassword -x proxy_name:proxy_port https://thelinuxcluster.com

8 Sending Email

% curl --url "smtps"//smtp.thelinuxcluster.com:465: --ssl-reqd --mail-from "sender@thelinuxcluster.com" --mail-rcpt "receiver@thelinuxcluster.com" --upload-file maincontent.txt --user "sender@thelinuxcluster.com:password" --insecure

References:

  1. Learn to use CURL command with examples
  2. Curl command in Linux with Examples

by

Finding physical cpus, cores and logical cpus

Number of Active Physical Processors

% grep physical.id /proc/cpuinfo | sort -u | wc -l
8

Number of cores per CPU

% grep cpu.cores /proc/cpuinfo | sort -u
cpu cores       : 26

Therefore the total number of cores is
8×26 = 208 cores

Number of Logical Processors

The number of cores seen by the Linux. Since the Server has switched on Hyperthreading 

% grep processor /proc/cpuinfo | wc -l
416

References:

  1. How to find the number of physical cpus, cpu cores, and logical cpus
by

Finding Top Processes using Highest Memory and CPU Usage in Linux

Read this Article from Find Top Running Processes by Highest Memory and CPU Usage in Linux. This is a quick way to view processes that consumed the largest RAM and CPU

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
   PID   PPID CMD                         %MEM %CPU
414699 414695 /usr/local/ansys_inc/v201/f 20.4 98.8
 30371      1 /usr/local/pbsworks/pbs_acc  0.2  1.0
 32241      1 /usr/local/pbsworks/pbs_acc  0.2  4.0
 30222      1 /usr/local/pbsworks/pbs_acc  0.2  0.6
  7191      1 /usr/local/pbsworks/dm_exec  0.1  0.8
 30595      1 /usr/local/pbsworks/pbs_acc  0.1  3.1
 30013      1 /usr/local/pbsworks/pbs_acc  0.1  0.3
 29602  29599 nginx: worker process        0.1  0.2
 29601  29599 nginx: worker process        0.1  0.3

The -o is to specify the output format. The -e is to select all processes. In order to sort in descending format, it hsould be –sort=–%mem

Interesting.