Compiling Singularity-CE-3.10.0 on Rocky Linux 8

The Official Documentation can be found at https://sylabs.io/guides/3.0/user-guide/installation.html

Prerequisites 1 – Go

Go to the Download Page https://go.dev/dl/ to download the Linux Version.

Extract the archive you downloaded into /usr/local, creating a Go tree in /usr/local/go.

This step below will remove a previous installation at /usr/local/go, if any, prior to extracting. Please back up any data before proceeding.

% rm -rf /usr/local/go && tar -C /usr/local -xzf go1.18.3.linux-amd64.tar.gz

Add /usr/local/go/bin to the PATH environment variable. You can do this by adding the following line to your $HOME/.profile or /etc/profile (for a system-wide installation):

export PATH=$PATH:/usr/local/go/bin

Verify the Installation with the command

% go version

Compiling Singularity

To download Singularity, do visit the download site. Singularity uses a build system called makeit. mconfig is called to generate Makefile and them make is used to compile and install

% git clone https://github.com/sylabs/singularity.git --recurse-submodules
% cd singularity
% ./mconfig --prefix=/usr/local/singularity-ce-3.10.0
% cd builddir
% make
% make install
.....
.....
 checking: header linux/securebits.h... yes
 checking: header linux/capability.h... yes
 checking: libseccomp+headers... yes
 checking: conmon source... no

conmon source not found

Unless you are building --without-conmon you must 'git clone --recurse-submodules'
or 'git submodule update --init'.

You may want to install glib2-devel, delete the singularity directory and make the singularity again.

% dnf install glib2-devel

Remote the cloned singularity directory and git clone again

% rm -Rv --force singularity
% git clone https://github.com/sylabs/singularity.git --recurse-submodules

Recompile with mconfig again. If successful, it should be something like

=> generating fragments ...
=> building Makefile ...
=> generating singularity-ce.spec ...
=> project singularity-ce setup with :
    - host arch: x86_64
    - host wordsize: 64-bit
    - host C compiler: cc
    - host Go compiler: /usr/local/go-1.18.3/bin/go
    - host system: unix
      ---
    - target arch: x86_64
    - target wordsize: 64-bit
    - target C compiler: cc
      ---
    - config profile: release
      ---
    - SUID install: yes
    - Network plugins: yes
    - seccomp support: yes
    - Build conmon: yes
      ---
    - verbose: no
      ---
    - cryptsetup: /usr/sbin/cryptsetup
      ---
    - version: 3.10.0+21-g1b1a05ff8
% cd builddir
% make
% make install

Testing

As long as you see a cow your installation is working properly…….

% singularity run library://godlovedc/funny/lolcow
< Exercise caution in your daily affairs. >
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

References:

  1. https://sylabs.io/guides/3.0/user-guide/installation.html
  2. https://github.com/NIH-HPC/Singularity-Tutorial
  3. Compiling Singularity-CE-3.9.2 on CentOS-7

Basic Usage of Singularity (Part 3)

Building a Basic Container

The information can be taken from https://github.com/NIH-HPC/Singularity-Tutorial. I’m just documenting what I understand step-by-step

In Order for you to build a Container, you have to use a build command. The build command does the following

  • Installs an OS
  • Setup the Container’s Environment
  • Install the apps required

Singularity Flow – Standard Development Cycle

  • Create a writable container (called a sandbox)
  • Shell into the container with the –writable option and tinker with it interactively
  • Record changes that we like in our definition file
  • Rebuild the container from the definition file if we break it
  • Rinse and Repeat
  • Rebuild the container from the final definition file as a read-only singularity image format (SIF) image for use in production

Let’s Build – Using the examples.

You can git clone Singularity Download Page and the examples is provided there.

% git clone https://github.com/sylabs/singularity.git
% mkdir ~lolcow
% cd singularity/examples/debian/Singularity ~/lolcow/lolcow.def
% cd ~/lolcow
% vim lolcow.def
BootStrap: debootstrap
OSVersion: stable
MirrorURL: http://ftp.us.debian.org/debian/

%runscript
    echo "This is what happens when you run the container..."

%post
    echo "Hello from inside the container"
    apt-get -y --allow-unauthenticated install vim

You can find more information on the Singularity Definition Files

Developing a new container

To build a container, you will need sudo privilege or root access

% sudo singularity build --sandbox lolcow lolcow.def
  • Container: lolcow
  • Definition File: lolcow.def
  • Build a Container for Development Purposes: –sandbox

After firing the command, you will have a basic Debian contained saved in a local directory called lolcow

Explore and modify the container

In order to obtain root privileges within the root container, you will need root at the Host System

% sudo singularity shell --writable lolcow

The –writable option allows us to modify the container. The corresponding changes will be saved into the container and persist across uses.

Installing Packages in Singularity

Once inside the shell, you can modify the container and it will be saved persistently.

Singularity > apt-get install -y fortune cowsay lolcat

you have to PATH it so that container can find the binary

Singularity > export PATH=$PATH:/usr/games
Singularity> fortune | cowsay | lolcat
 ________________________________________
/ Q: Why haven't you graduated yet? A:   \
| Well, Dad, I could have finished years |
| ago, but I wanted                      |
|                                        |
\ my dissertation to rhyme.              /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Building Production Grade

 If you ultimately want all of these changes to be reflected in your definition file. Let’s update our definition file

% nano lolcow.def
BootStrap: debootstrap
OSVersion: stable
MirrorURL: http://ftp.us.debian.org/debian/

%runscript
    echo "This is what happens when you run the container..."

%post
    echo "Hello from inside the container"
    apt-get update
    apt-get -y install fortune cowsay lolcat

%environment
    export PATH=$PATH:/usr/games

To rebuild the image from the definition, make sure you have the necessary yum package in your host.

# yum install debootstrap
# yum install squashfs-tools

If you are having further issues, make sure your hosts has the prerequistics packages installed. It can be found at https://github.com/NIH-HPC/Singularity-Tutorial/tree/master/01-installation

Building the Container

# singularity build lolcow.sif lolcow.def
.....
.....
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: lolcow.sif

References:

Basic Usage of Singularity (Part 2)

The information can be taken from https://github.com/NIH-HPC/Singularity-Tutorial. I’m just documenting what I understand step-by-step

EXEC Command

Using the exec command, we can run commands within the container from the host system.

[user1@node ~]$ singularity exec lolcow_latest.sif cowsay 'How is today?'
 _______________
< How is today? >
 ---------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

RUN Command

Sometimes you can run a container

 [user1@node ~]$ singularity run lolcow_latest.sif
 _____________________________________
/ Fine day to work off excess energy. \
\ Steal something heavy.              /
 -------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

A special script “runscript” is activated when the “run” command is exercised. To have a closer look

[user1@node ~]$ singularity inspect --runscript lolcow_latest.sif
#!/bin/sh

    fortune | cowsay | lolcat

runscript consists of three simple commands with the output of each command piped to the subsequent command. Altenatively, you can do a

[user1@node ~]$ ./lolcow_latest.sif

Pipes and Redirection

Singularity allows piping services so that the host system can interact with the container. Here we are executing a command in the container and piping the output out into a file called output.txt

[user1@node ~]$ singularity exec lolcow_latest.sif cowsay moo > output.txt

You will notice that the output has been pushed to output.txt. Make sure you create output.txt prior

[user1@node ~]$ vim output.txt

Inside output.txt

 How are you >
 -------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Let’s say we create a file called HelloThere and put in a text called “Shame On You”. The command is then fed into cowsay.

% cat HelloThere | singularity exec lolcow_latest.sif cowsay -n
______________
< Shame on You >
 --------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

References:

  1. https://github.com/NIH-HPC/Singularity-Tutorial

Basic Usage of Singularity (Part 1)

The article is taken from https://github.com/NIH-HPC/Singularity-Tutorial. I’m personally learning much from the systematic rich contents and below are my learning points……

Downloading Singularity

https://github.com/sylabs/singularity

Popular Sites

There are popular sites for Singularity Download that has pre-build containers

Downloading the Containers

% singularity pull library://godlovedc/funny/lolcow
INFO:    Downloading library image
89.2MiB / 89.2MiB [========================================================================================================================================================] 100 % 5.4 MiB/s 0s
WARNING: integrity: signature not found for object group 1
WARNING: Skipping container verification

Singularity File

The Singularity File has a .sif extension something like this

lolcow_latest.sif

Entering the Shell of Singularity

% singularity shell lolcow_latest.sif
Singularity> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

You will notice the container is running Ubuntu…… although the Host OS could be different

Interesting Observations

a. The user remains the same inside and outside of the container.

Singularity> whoami
admin

b. The hostname remains the same inside and outside the container

Singularity> hostname
node1

c. Running the application within the containers

Singularity> cowsay moo
 _____
< moo >
 -----
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

d. Exiting the Application

Singularity > exit

References:

  1. https://github.com/NIH-HPC/Singularity-Tutorial
  2. https://github.com/sylabs/singularity

Compiling Singularity-CE-3.9.2 on CentOS-7

The Official Documentation can be found at https://sylabs.io/guides/3.0/user-guide/installation.html

Prerequisites 1 – Yum

If you are using CentOS, you may want to put the necessary libraries and dependencies in first.

yum install -y && \
yum groupinstall -y 'Development Tools' && \
yum install -y \
openssl-devel \
libuuid-devel \
libseccomp-devel \
wget \
squashfs-tools

Prerequisites 2 – Go

Go to the Download Page https://go.dev/dl/ to download the Linux Version.

Extract the archive you downloaded into /usr/local, creating a Go tree in /usr/local/go.

This step below will remove a previous installation at /usr/local/go, if any, prior to extracting. Please back up any data before proceeding.

% rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.5.linux-amd64.tar.gz

Add /usr/local/go/bin to the PATH environment variable. You can do this by adding the following line to your $HOME/.profile or /etc/profile (for a system-wide installation):

export PATH=$PATH:/usr/local/go/bin

Verify the Installation with the command

% go version

Compiling Singularity

To download Singularity, do visit the download site. Singularity uses a build system called makeit. mconfig is called to generate Makefile and them make is used to compile and install

% cd singularity
% ./mconfig
 --prefix=/usr/local/singularity-ce-3.9.2
% cd build
% make
% make install

Source bash-completion file

To enjoy bash completion with Singularity commands and options, source the bash-completion file. you need to source the appropriate file and add it to the bash-completion directory in /etc so that it will be sourced automatically when you start another shell.

% . /usr/local/singularity-ce-3.9.2/etc/bash_completion.d/singularity
% cp /usr/local/singularity-ce-3.9.2/etc/bash_completion.d/singularity /etc/bash_completion.d/

Testing

As long as you see a cow your installation is working properly…….

singularity run library://godlovedc/funny/lolcow
/ Q: What is the difference between a \
\ duck? A: One leg is both the same.  /
 -------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

References

  1. https://sylabs.io/guides/3.0/user-guide/installation.html
  2. https://github.com/NIH-HPC/Singularity-Tutorial