System Administration

by

General Linux OS Tuning for AMD EPYC

Step 1: Turn off swap
Turn off swap to prevent accidental swapping. Do not that disabling swap without sufficient memory can have undesired effects

swapoff -a

Step 2: Turn off NUMA balancing
NUMA balancing can have undesired effects and since it is possible to bind the ranks and memory in HPC, this setting is not needed

echo 0 > /proc/sys/kernel/numa_balancing

Step 3: Disable ASLR (Address Space Layout Ranomization) is a security feature used to prevent the exploitation of memory vulnerabilities

echo 0 > /proc/sys/kernel/randomize_va_space

Step 4: Set CPU governor to performance and disable cc6. Setting the CPU perfomance to governor to perfomrnaces ensures max performances at all times. Disabling cc6 ensures that deeper CPU sleep states are not entered.

cpupower frequency-set -g performance
Setting cpu: 0
Setting cpu: 1
.....
.....
cpupower idle-set -d 2
Idlestate 2 disabled on CPU 0
Idlestate 2 disabled on CPU 1
Idlestate 2 disabled on CPU 2
.....
.....

References:

  1. Tuning Guard for AMD EPYC (pdf)
by

Getting Useful Information on CPU and Configuration

Point 1. lscpu

To install

yum install util-linux

lscpu – (Print out information about CPU and its configuration)

[user1@myheadnode1 ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
Stepping: 4
CPU MHz: 3200.000
BogoMIPS: 6400.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
Flags: fpu .................

Point 2: hwloc-ls

To install hwloc-ls

yum install hwloc

hwloc – (Prints out useful information about the NUMA locality of devices and general hardware locality information)

[user1@myheadnode1 ~]# hwloc-ls
Machine (544GB total)
NUMANode L#0 (P#0 256GB)
Package L#0 + L3 L#0 (25MB)
L2 L#0 (1024KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#16)
L2 L#1 (1024KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#1)
PU L#3 (P#17)
L2 L#2 (1024KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
PU L#4 (P#2)
PU L#5 (P#18)
L2 L#3 (1024KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
PU L#6 (P#3)
PU L#7 (P#19)
L2 L#4 (1024KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
PU L#8 (P#4)
PU L#9 (P#20)
L2 L#5 (1024KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
PU L#10 (P#5)
PU L#11 (P#21)
L2 L#6 (1024KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
PU L#12 (P#6)
PU L#13 (P#22)
L2 L#7 (1024KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
PU L#14 (P#7)
PU L#15 (P#23)
.....
.....
.....

Point 3 – Check whether the Boost is on for AMD

Print out if CPU boost is on or off

cat /sys/devices/system/cpu/cpufreq/boost
1

References:

  1. Tuning Guard for AMD EPYC (pdf)
by

Turning ksm and ksmtuned off

In this blog, I will write on how to turn off KSM and ksmtuned since I do not need these services and save some unnecessary swapping activities on the disk.

What is KSM?

According to RedHat Site (8.4. KERNEL SAME-PAGE MERGING (KSM)),
Kernel same-page Merging (KSM), used by the KVM hypervisor, allows KVM guests to share identical memory pages. These shared pages are usually common libraries or other identical, high-use data. KSM allows for greater guest density of identical or similar guest operating systems by avoiding memory duplication……

KSM is a Linux feature which uses this concept in reverse. KSM enables the kernel to examine two or more already running programs and compare their memory. If any memory regions or pages are identical, KSM reduces multiple identical memory pages to a single page……

8.4.4 Kernel same-page merging (KSM) has a performance overhead which may be too large for certain environments or host systems. KSM may also introduce side channels that could be potentially used to leak information across guests. If this is a concern, KSM can be disabled on per-guest basis.

Deactivating KSM

# systemctl stop ksmtuned
Stopping ksmtuned:                                         [  OK  ]
# systemctl stop ksm
Stopping ksm:                                              [  OK  ]

To permanently deactivate KSM with the systemctl commands

# systemctl disable ksm
# systemctl disable ksmtuned

When KSM is disabled, any memory pages that were shared prior to deactivating KSM are still shared. To delete all of the PageKSM in the system, use the following command:

# echo 2 >/sys/kernel/mm/ksm/run

After this is performed, the khugepaged daemon can rebuild transparent hugepages on the KVM guest physical memory. Using # echo 0 >/sys/kernel/mm/ksm/run stops KSM, but does not unshare all the previously created KSM pages (this is the same as the # systemctl stop ksmtuned command).

References:

  1. Redhat – 8.4. Kernel Same-Page Merging (KSM)
by

Using strace to detect df hanging issues on NFS

strace is a wonderful tool to trace system calls and signals

I was hanging issues whenever I do a “df” and I was curious which file system is calling issues

strace df
.....
.....
stat("/run/user/1304561586", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/17132623", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/17149581", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/run/user/1304565184", {st_mode=S_IFDIR|0700, st_size=60, ...}) = 0
stat("/scratch",

It is obvious that /scratch file hang immediately after being launched.

by

Basic GNU Screen Usage on CentOS

Introduction

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells. In other words, you can start any number of virtual terminals inside the session. The good thing is that processes running inside screen will continue to run even though the SSH session get disconnected.

GNU Screen Site

GNU Screen can be found on http://ftp.gnu.org/gnu/screen/

Source Code

You can get the source code from here

Using Screen

Screen can be easily installed on CentOS with just

# yum install screen

Naming a Named Session

You may be running many sessions and it is a good idea to name the session that you are starting.

screen -S your_preferred_screen_name

Listing running Screen Session

[user1@node1 ~]$ screen -ls
There is a screen on:
2109.myScreenA (Detached)
1 Socket in /var/run/screen/S-user1

Reattach to a Screen Session

To connect back to the screen, just type the numeric id of the screen

screen -r 2109

Detaching from a Screen Session

Inside the Virtual Session, you can detach the screen with the command

Ctrl-a + d

[Press ctrl with “a” and “d” together]
If you are already outside the virtual session, you can detach an active session by

screen -d 2109

Customised Screen

If you a looking at how to split Screen using screen. Here is a good visual guide.

To Terminate the Screen Session,

Enter into the Session screen

screen -r 2109
exit

 

User Guide

https://www.gnu.org/software/screen/manual/screen.html#Startup-Files

Links:

How To Use Linux Screen

by

Unable to open /dev/sdb with fdisk

Taken from my old blog Unable to open /dev/sdb with fdisk

Fdisk is a menu driven program for creation and manipulation of partition tables. The device is usually something like /dev/sda, /dev/sdb. A device name refers to the entire disks. /dev/sd? is the partition of the device. For example, /dev/sda1 refers to the first partition of the first device.

If you issued a command and you receive a corresponding message “unable to open /dev/sdb”

# fdisk /dev/sdb

 

Unable to open /dev/sdb

Linux is unable to locate or find the partition. One method to verify that it is so, do a listing of the

devices fdisk can see. In this example below, the partition has been created already.

# fdisk -l

 

Disk /dev/sdb: 2997.4 GB, 2997426536960 bytes
255 heads, 63 sectors/track, 364416 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      267349  2147480811   83  Linux

WARNING: The size of this disk is 3.0 TB (2997400633344 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Once you have verified the presence of the device, do a fdisk /dev/sdb again

 

by

xrdp_mm_process_login_response: login failed on CentOS 6

This post is taken from my old blog xrdp_mm_process_login_response: login failed

If you encountered this error xrdp_mm_process_login_response: login failed when you use the remote desktop connection to connection to a vnc session.

Even if you restart xrdp, the error still remain, the issue could be due to locked  X11 session that was created by xrdp.

To solve the issue, go to the/tmp/.X11-unix/ and find your X session and delete the session.

# cd /tmp/.X11-unix

Do a listing

# ls -l

Look at the session owned by you which you wished to delete

.....
.....
srwxrwxrwx 1 root      root  0 Jul  9  2012 X0
srwxrwxrwx 1 user1  users 0 Jan 25 09:13 X1
srwxrwxrwx 1 user2      users 0 Jul 10  2012 X10
srwxrwxrwx 1 user3     users 0 Feb 19 13:31 X11
srwxrwxrwx 1 user4  users 0 Nov 20 15:10 X12
srwxrwxrwx 1 user5     users 0 Jul 10  2012 X13
.....
.....

Delete the session……

If xrdp still fails, it seems that it is due to orphaned X–. Once xrdp hits an orphaned X– which may or may not be from other users, the error will still remain.

To see the orphaned X11 session, you can do a vncserver, which you will see something like that

# vncserver

 

Warning: Head-Node:1 is taken because of /tmp/.X11-unix/X1
Remove this file if there is no X server Head-Node:1

Delete all the orphaned X–
Restart the xrdp service and try the remote connection.

# service xrdp restart

If you are still having the issue, do look at alternative solution

  1. X Server — no display in range is available. xrdp_mm_process_login_response: login failed

 

by

Fixing Authentication is required to set the network proxy used for download packages on CentOS 6

This was an old post from my dated blogger Fixing Authentication is requried to set the network proxy used for download packages for CentOS 6

I encountered this pop-up error today when I was xrdp into my CentOS 6. The error was something like this

authentication is required to set the network proxy used for downloading packages.  
An application is attempting to perform an action that requires privileges.  
Authentication as the super user is required to perform this action" and asking  
for the root password



Non-root users

Step 1: Launch a Terminal Console and type

# gnome-session-properties

Step 2: Uncheck PackageKit Update Applet .

See pix below.

Root User

Step 1: Disabled /etc/yum/pluginconf.d

# vim /etc/yum/pluginconf.d/refresh-packagekit.conf
[main]
enabled=0

 

by

Making HTTP Server less vulnerable DOS Attack

**These steps were done on CentOS 6 Server

Step 1: Upgrade Apache HTTP to the latest version

# yum update httpd

Step 2: Edit the httpd.conf.

# vim /etc/httpd/conf/httpd.conf

Inside httpd.conf. Scroll all the way to LoadModule………. section and add the line

LoadModule reqtimeout_module modules/mod_reqtimeout.so

Step 3: Create a /etc/httpd/conf.d/reqtimeout.conf and put in the informationbelow

<ifmodule reqtimeout_module>
RequestReadTimeout header=10-20,minrate=500
RequestReadTimeout body=10,minrate=500
</ifmodule>

Step 4: To check whether the apache module is loaded, do

# apachectl -M

Or

apache2ctl -M

References:

  1. Mitigating WAS QID 150085 Slow HTTP POST Vulnerability on Apache

 

by

xRDP Connection Errors on CentOS 6

After yum install xrdp and starting the service, I encountered the error during remote desktop to the Linux Box.

connecting to sesman ip 127.0.0.1 port 3350
sesman connect ok
sending login info to session manager, please wait...
xrdp_mm_process_login_reponse: login successful for display
started connecting
connecting to 127.0.0.1 5910
error - problem connecting

At the /var/log/xrdp-sesman.log

......
[20141118-23:53:40] [ERROR] X server for display 10 startup timeout
[20141118-23:53:40] [INFO ] starting xrdp-sessvc - xpid=2998 - wmpid=2997
[20141118-23:53:40] [ERROR] X server for display 10 startup timeout
[20141118-23:53:40] [ERROR] another Xserver is already active on display 10
[20141118-23:53:40] [DEBUG] aborting connection...
[20141118-23:53:40] [INFO ] ++ terminated session:  username root, display :10.0
.....

I have installed the necessary GNOME Desktop packages. Installing GNOME Desktop on CentOS 6 on a console before installing xrdp.

But the solution is quite simple. You need to install the tigervnc-server package and just the tigervnc only.

# yum install tigervnc-server

Restart the xrdp again.

# service xrdp restart