Application

by

Resolving GNU MP not found on CentOS 7

If you are installing package like BiocManager::install(“scDblFinder”)

BiocManager::install("scDblFinder")

You may encounters error like 

configure: error: GNU MP not found, or not 4.1.4 or up, see http://gmplib.org

The fix is obvious as seen on error message. If you are using CentOS 7, you can easily fix it via Yum

% yum install gmp-devel
by

Compiling R-4.1.0 with GNU

The R Project for Statistical Computing

Prerequisites

gnu-6.5
m4-1.4.18
gmp-6.1.0
mpfr-3.1.4
mpc-1.0.3
isl-0.18
gsl-2.1

Compiling PCRE is important or you will face an error like

configure: error: PCRE2 library and headers are required, or use --with-pcre1 and PCRE >= 8.32 with UTF-8 support

After you have compile PCRE, you can proceed with the compilation of R-4.1.0

% ./configure --prefix=/usr/local/R-4.1.0 --with-pcre1=/usr/local/pcre-8.42 --with-blas --with-lapack --enable-R-shlib

If there are no issues…. 

R is now configured for x86_64-pc-linux-gnu

  Source directory:            .
  Installation directory:      /usr/local/R-4.1.0

  C compiler:                  gcc  -g -O2
  Fortran fixed-form compiler: gfortran  -g -O2

  Default C++ compiler:        g++ -std=gnu++14  -g -O2
  C++11 compiler:              g++ -std=gnu++11  -g -O2
  C++14 compiler:              g++ -std=gnu++14  -g -O2
  C++17 compiler:
  C++20 compiler:
  Fortran free-form compiler:  gfortran  -g -O2
  Obj-C compiler:              gcc -g -O2 -fobjc-exceptions

  Interfaces supported:        X11
  External libraries:          pcre1, readline, BLAS(generic), LAPACK(generic), curl
  Additional capabilities:     PNG, JPEG, NLS, ICU
  Options enabled:             shared R library, R profiling

  Capabilities skipped:        TIFF, cairo
  Options not enabled:         shared BLAS, memory profiling

  Recommended packages:        yes

Make and Make Install the Files

% make -j 8
.....
gcc -I"/home/user1/Downloads/R-4.1.0/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c anova.c -o anova.o
gcc -I"/home/user1/Downloads/R-4.1.0/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c anovapred.c -o anovapred.o
gcc -I"/home/user1/Downloads/R-4.1.0/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c branch.c -o branch.o
gcc -I"/home/user1/Downloads/R-4.1.0/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c bsplit.c -o bsplit.o
gcc -I"/home/user1/Downloads/R-4.1.0/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c choose_surg.c -o choose_surg.o
g
.....
% make install

References:

  1. Compiling R by Toby Dylan
by

Basic CURL Commands

curl is a command line tool to transfer data to or from a server. It is able to use any of the supported protocols like HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP or FILE. This tool is very useful for automation, since it is designed to work without user interaction. Furthermore, curl can transfer multiple file at once.

Basic Single URL Usage

% curl https://thelinuxcluster.com/

2a. Save the Download File with a preferred file name

Save the Download File on the local machine with the name provided with the parameter.

% curl -o test.o https://thelinuxcluster.com/test.output

2b. Save the Download File

% curl -O https://thelinuxcluster.com/test.output

2c. Download Multiple Files. Just Multiple -O

% curl -O https://thelinuxcluster.com/CentOS1.iso -O https://thelinuxcluster.com/CentOS2.iso -O https://thelinuxcluster.com/CentOS3.iso

3a. Display a Progress Meter

% curl --progress-bar -o test.o https://thelinuxcluster.com/test.output

3b. Do not display a Progressive Bar

% curl --silent -o test.o https://thelinuxcluster.com/test.output

4 Limit Rate of Data Transfer

% curl --limit-rate 1000K -o test.o https://thelinuxcluster.com/CentOS1.iso

5a Uploading a File to the FTP Server

% curl -u username:userpassword -T myfile ftp://ftp.thelinuxcluster.com/

5b. Appending the File to the FTP Server

% curl -u username:userpassword -a -T myfile ftp://ftp.thelinuxcluster.com/

5c Downloading the File to the File Server

% curl ftp:/ftp.thelinuxcluster.com/CentOS79.iso --user username:userpassword -o myCentOS79.iso

6a. Verifying SSL Certificate

% curl --cacert server.crt https://thesupersecureserver.com

6b. Ignoring SSL Certificate

% curl -k https://thesupersecureserver.com/

7a Proxy Server

% curl -x proxy_name:proxy_port https://thelinuxcluster.com

7b Proxy Server which requires authentication

% curl --user username:userpassword -x proxy_name:proxy_port https://thelinuxcluster.com

8 Sending Email

% curl --url "smtps"//smtp.thelinuxcluster.com:465: --ssl-reqd --mail-from "sender@thelinuxcluster.com" --mail-rcpt "receiver@thelinuxcluster.com" --upload-file maincontent.txt --user "sender@thelinuxcluster.com:password" --insecure

References:

  1. Learn to use CURL command with examples
  2. Curl command in Linux with Examples

by

Creating Virtual Environment with Python using venv

Virtual Environment and Packages

Sometimes there is a need to create virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages as there is sometimes a need for specific version of python library. Here come venv

Creating Virtual Environments 

% module load python/3/intel/2020
% python3 -m venv myenv

This will create a myenv directory if it doesn’t exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files. 

% source myenv/bin/activate

Activating the virtual environment will change your shell’s prompt to show what virtual environment you’re using,

(tutorial-env) [user1@hpc-node1 bin]$ python
Python 3.7.4 (default, Nov 22 2019, 21:31:39)
[GCC 7.3.0] :: Intel(R) Corporation on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distibution
>>> import sys
>>> sys.path
['', '/usr/local/intelpython3/lib/python37.zip', ....., '.....', '/myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages']
>>>

Managing Packages with pip

You can install, upgrade and remove using a program called pip. By default, pip will install packages from the Python Package Index (http://pypi.org).

You can install the latest version of a package by specifying a package’s name:

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install novas
Collecting novas
  Downloading https://files.pythonhosted.org/packages/42/95/a05bc35cb119925e10f9faa8a2bd17020b0a585744a38921a709acdd9a14/novas-3.1.1.5.tar.gz (135kB)
    100% |████████████████████████████████| 143kB 4.6MB/s
Installing collected packages: novas
  Running setup.py install for novas ... done
Successfully installed novas-3.1.1.5

You can also install specific version by using the command 

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install requests==2.6.0
Collecting requests==2.6.0
  Downloading https://files.pythonhosted.org/packages/73/63/b0729be549494a3e31316437053bc4e0a8bb71a07a6ee6059434b8f1cd5f/requests-2.6.0-py2.py3-none-any.whl (469kB)
    100% |████████████████████████████████| 471kB 4.3MB/s
Installing collected packages: requests
Successfully installed requests-2.6.0

You can also upgrade the package by using the command

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install --upgrade request    s
Collecting requests
  Downloading https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50    280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl     (61kB)
    100% |████████████████████████████████| 61kB 3.4MB/s
Collecting chardet<5,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87    d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (1    78kB)
    100% |████████████████████████████████| 184kB 5.3MB/s
Collecting urllib3<1.27,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/0c/cd/1e2ec680ec7b09846dc    6e605f5a7709dfb9d7128e51a026e7154e18a234e/urllib3-1.26.5-py2.py3-none-any.whl (    138kB)
    100% |████████████████████████████████| 143kB 5.9MB/s
Collecting idna<3,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6    f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 4.3MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec    9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.wh    l (145kB)
    100% |████████████████████████████████| 153kB 5.1MB/s
Installing collected packages: chardet, urllib3, idna, certifi, requests
  Found existing installation: requests 2.6.0
    Uninstalling requests-2.6.0:
      Successfully uninstalled requests-2.6.0
Successfully installed certifi-2021.5.30 chardet-4.0.0 idna-2.10 requests-2.25.    1 urllib3-1.26.5

You can display all the packages in the environment in virtual environment

(tutorial-env) [user1@hpc-node1 bin]$ pip list
Package    Version
---------- ---------
certifi    2021.5.30
chardet    4.0.0
idna       2.10
novas      3.1.1.5
pip        19.0.3
requests   2.25.1
setuptools 40.8.0
urllib3    1.26.5

pip freeze can be used to produce a similar list of installed packages and formatted in a output file that pip install can read from

(tutorial-env) [user1@hpc-node1 bin]$ pip freeze > requirements.txt
(tutorial-env) [user1@hpc-node1 bin]$ cat requirements.txt
certifi==2021.5.30
chardet==4.0.0
idna==2.10
novas==3.1.1.5
requests==2.25.1
urllib3==1.26.5

Install all the necessary packages with install -r from requirements.txt

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install -r requirements.txt
Requirement already satisfied: certifi==2021.5.30 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (2021.5.30)
Requirement already satisfied: chardet==4.0.0 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (4.0.0)
Requirement already satisfied: idna==2.10 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (2.10)
Requirement already satisfied: novas==3.1.1.5 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (3.1.1.5)
Requirement already satisfied: requests==2.25.1 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (2.25.1)
Requirement already satisfied: urllib3==1.26.5 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.26.5)

References:

by

RELION – Performance Benchmark and Profiling

What is RELION?

RELION (REgularized LIkelihood OptimizatioN) is an open-source program for the refinement of macromolecular structures by single-particle analysis of electron cryomicroscopy (cryo-EM) data

RELION (REgularized LIkelihood OptimizatioN) implements an empirical Bayesian approach for analysis of electron cryo-microscopy (Cryo-EM)

RELION provides refinement methods of singular or multiple 3D reconstructions as well as 2D class averages

RELION is an important tool in the study of living cells

HPC-AI Advisory Council

Performance Analysis Summary

(from Article See RELION – Performance Benchmark and Profiling)

RELION performance testing

  • Pool size 4,8,16 gave best performance on 16,24,32 nodes
  • SHARP In-Network Computing reduces MPI time by 13% and increase overall application performance by 5
  • Performance advantages increases with system size, up to 32 nodes were tested

RELION Profile

  • Rank #0 does not perform computation
  • Mostly MPI_Barrier (70%)
  • Ring communication matrix

References:

by

Compiling GAMESS-v2020.2 with Intel MPI

GAMESS Download Site can be found at https://www.msg.chem.iastate.edu/GAMESS/download/dist.source.shtml

Compiling GAMESS

% tar -zxvf gamess-current.tar.gz
% cd gamess
% ./config

You have to answer the following question on

  • Machine Type? – I chose “linux64“
  • GAMESS directory? -I chose “/usr/local/gamess“
  • GAMESS Build Directory – I chose “/usr/local/gamess“
  • Version? [00] – I chose default [00]
  • Choice of Fortran Compilers – I chose “ifort”
  • Version Number of ifort – I chose “18” (You can check by issuing the command ifort -V)
  • Standard Math Library – I chose “mkl”
  • Path of MKL – I chose “/usr/local/intel/2018u3/compilers_and_libraries_2018.3.222/linux/mkl“
  • Type “Proceed” next
  • Communication Library – I chose “mpi” (I’m using Infiniband)
  • Enter MPI Library – I chose “impi“
  • Enter Location of impi – I chose “/usr/local/intel/2018u3/impi/2018.3.222“
  • Build experimental support of LibXC – I chose “no“
  • Build Beta Version of Active Space CCT3 and CCSD3A – I chose “no“
  • Build LIBCCHEM – I chose “no“
  • Build GAMESS with OpenMP thread support – I chose “yes”

Once done, you should see

Your configuration for GAMESS compilation is now in
     /usr/local/gamess/install.info
Now, please follow the directions in
     /usr/local/gamess/machines/readme.unix

Compiling ddi

Edit DDI Node Sizes by editing /usr/local/gamess/ddi/compddi
Look at Line 90 and 91. You may want to edit MAXCPUS and MAXNODES. Once done, you can compile ddi

% ./compddi >& compddi.log &

Compiling GAMESS

The compilation will take a while. So relax…..

% ./compall >& compall.log &

Linking Executable Form of GAMESS with the command

./lked gamess 01 >& lked.log &

Edit the Scratch Directory setting at rungms

% vim rungms
set SCR=/scratch/$USER
by

Compiling flac-1.3.3 with GNU 6.5

If you are hoping to compile flac-1.3.3 with libogg-1.3.4, do the following

Step 1: Download latest libogg from Xiph.org

Step 2: Untar and Compile the libogg

% tar -zxvf libogg-1.3.4.tar.gz% cd libogg-1.3.4
% ./configure --prefix=/usr/local/libogg-1.3.4
% make && make install

Step 3: Download flac-1.3.3 from https://github.com/xiph/flac

Step 4: Untar and Compile flac-1.3.3 with libogg-1.3.4

% git clone https://github.com/xiph/flac.git
% cd flac
% ./autogen.sh
% ./configure --prefix=/usr/local/flac-1.3.3 --with-ogg-libraries=/usr/local/libogg-1.3.4/lib --with-ogg-includes=/usr/local/libogg-1.3.4/include/

-=-=-=-=-=-=-=-=-=-= Configuration Complete =-=-=-=-=-=-=-=-=-=-

  Configuration summary :

    FLAC version : ............................ 1.3.3

    Host CPU : ................................ x86_64
    Host Vendor : ............................. unknown
    Host OS : ................................. linux-gnu

    Compiler is GCC : ......................... yes
    GCC version : ............................. 4.8.5
    Compiler is Clang : ....................... no
    SSE optimizations : ....................... yes
    Asm optimizations : ....................... yes
    Ogg/FLAC support : ........................ yes
    Stack protector  : ........................ yes
    Fuzzing support (Clang only) : ............ no

% make && make install