Month: February 2010

by

Overview of MAUI Scheduler Commands

  MAUI is an open source job scheduler for clusters and supercomputers. It is an optimized, configurable tool capable of supporting an array of scheduling policies, dynamic priorities, extensive reservations, and fairshare capabilities.

This Blog Entry attempt to capture the essence of MAUI and some of the more commonly used commands and configuration.

To download MAUI Scheduler, go to Maui Cluster Scheduler. To download the MAUI Documentation, proceed to Cluster Resources Documentation

Useful commands for MAUI

1. Configuring MAUI Scheduler

  1. schedctl -R command can be used to reconfigure the scheduler at any time, forcing it to re-read all config files before continuing.
  2. Shut-down MAUI Scheduler 
    # schedctl -k
  3. Stop maui scheduling 
    # schedctl -s
  4. maui will resume scheduling immediately 
    # schedctl -r

 2. Status Commands

 Maui provides an array of commands to organize and present information about the current state and historical statistics of the scheduler, jobs, resources, users, accounts, etc. The following commands are taken from Cluster Resources and reproduce here

checkjob -> display job state, resource requirements, environment, constraints,
credentials, history, allocated resources, and resource utilization
checknode -> Displays state information and statistics for the specified node.
diagnose -j -> display summarized job information and any unexpected state
diagnose -n -> display summarized node information and any unexpected state
diagnose -p -> display summarized job priority information
diagnose -r -> display summarized reservation information
showgrid -> display various aspects of scheduling performance across a job duration/job size matrix
showq -> display various views of currently queued active, idle, and non-eligible jobs
showstat -f -> display historical fairshare usage on a per credential basis
showstat -g -> display current and historical usage on a per group basis
showstat -u -> display current and historical usage on a per user basis
showstat -v -> display high level current and historical scheduling statistics

3. Job Management Commands

Maui shares job management tasks with the resource manager. The commands below the available job management commands

canceljob   -> cancel existing job
releasehold [-a]  -> remove job holds or defers
runjob   -> start job immediately if possible
sethold   -> set hold on job
setqos   -> set/modify QoS of existing job
setspri   -> adjust job/system priority of job

4. Reservation Management Commands

Maui exclusively controls and manages all advance reservation features including both standing and administrative reservations

diagnose -r -> display summarized reservation information and any unexpected state
releaseres -> remove reservations
setres -> immediately create an administrative reservation
showres -> display information regarding location and state of reservations

5. Policy/Config Management Commands

Maui allows dynamic modification of most scheduling parameters allowing new scheduling policies, algorithms, constraints, and permissions to be set at any time.

changeparam  -> immediately change parameter value
schedctl  -> control scheduling behavior (i.e., stop/start scheduling, recycle, shutdown, etc.)
showconfig ->  display settings of all configuration parameters

6. End User Commands

canceljob ->  cancel existing job
checkjob  -> display job state, resource requirements, environment, constraints, credentials, history, allocated resources, and resource utilization
showbf  -> show resource availability for jobs with specific resource requirements
showq ->  display detailed prioritized list of active and idle jobs
showstart ->  show estimated start time of idle jobs
showstats  -> show detailed usage statistics for users, groups, and accounts which the end user has access to
by

A Brief Introduction to Linux Virtual Cluster

According to the project website, Linux Virtual Server (LVS) is a highly scalable and highly available server built on a cluster of real servers. The architecture of server cluster is fully transparent to end users, and the users interact with the cluster system as if it were only a single high-performance virtual server.

Much of the information is taken from the book “The Linux Enterprise Cluster” by Karl Cooper and the Project Website.

This diagram is taken from the project website which explain very clearly the purpose of this project.

The Linux Virtual Server  (LVS) accepts all incoming client computer requests for services and decides which one in the cluster nodes will reply to the client. Some naming convention used by the LVS community

  1. Real Server refers to the nodes inside the an LVS Cluster
  2. Client Computer refers to computer outside the LVS Cluster
  3. Virtual IP (VIP) address refers to the IP address the Director uses to offer services to client computer. A single LVS Director can have multiple VIPS offering different services to client computerx. This is the only IP that the client computer needs to know to access to.
  4. Real IP (RIP) address refers to the IP Address used on the cluster node. Only the LVS Director is required to know the IP Addresses of this node
  5. Director IP (DIP) address refers to the IP address the LVS Director uses to connect to the RIP network. As requests from the client computer comes, they are forwarded to the client PCs. the VIP and DIP can be on the same NIC
  6. Client IP Address (CIP) address refers to the IP address of the client pc.

 

A. Types of LVS Clusters

The types of LVS Clusters are usually described by the type of forwarding method, the LVS Director uses to relay incoming requests to the nodes inside the cluster

  1.  Network Address Translation (LVS-NAT)
  2. Direct routing (LVS-DR)
  3. IP Tunnelling (LVS-TUN)

 According to the Book “The Linux Enterprise Cluster by Karl Cooper”, the best forwarding method to use with a Linux Enterprise Cluster is a LVS-DR. The easier to build is LVS-NAT. The LVS-TUN is generally not in use for mission critical applications and mentioned for the sake of competencies.

  

A1. LVS-NAT Translation

 

 In a LVS-NAT setup, the Director uses the Linux kernel’s ability (from the kernel’s filter code) to translate IP Addresses and ports as packets pass through the kernel.

From the diagram above, the client send a request and is sent to the Director on its VIP. The Director redirects the request to the RIP of the Cluster Node. The Cluster Node reply back via its RIP to the Director. The Director convert the cluster node RIP into the VIP owned by the Director and return reply to the client.

Some basic notes on the LVS-NAT

  1. The Director intercepts all communication from clients to the cluster nodes
  2. The cluster nodes uses the Director DIP as their default gateway for reply packets to the client computers
  3. The Director can remap network port numbers
  4. Any operating system can be used inside the cluster
  5. Network or the Director may be a bottleneck. It is mentioned that a 400Mhz can saturate a 100MB connection
  6. It may be difficult to administer the cluster node as the administrator must enter the cluster node via the Director. Of course you can do a bit of network and firewall configuration to circumvent the limitation

 

A2 LVS-DR (Direct Routing)

s

In a LVS-DR setup, the Director for towards all the incoming requests to the nodes inside the cluster, but the nodes inside the cluster send their replies directly back to the client computer.

From the diagram, the client send a request and is send to the Director on its VIP. The Director redirects the request to the RIP of the Cluster Node. The cluster node reply back directly to the client and the packet uses the VIP as  its source IP Addresses. The Client is fooled into thinking it is talking to a single computer (Director)

Some Basic properties of the LVS-DR

  1. The cluster nodes must be on the same segment as the Director
  2. The Director intercepts inbound communication but not outbound communication between clients and the real servers
  3. The cluster node do not use the Director as the default gateway for reply packets to the client
  4. The Director cannot remap network port number
  5. Most operating systems can be used on the real server inside the cluster. However the Operating System must be capable of configuring the network interface to avoid replying to ARP broadcasts.
  6. If the Director fails, the cluster nodes becomes distributed servers each with its own IP Addresses. You can “save” the situation by using Round-Robin DNS to hand out the RIP addresses for each cluster node. Alternatively, you can “save” the situation by asking the users to connect to the cluster node directly.

 

A3 LVS-TUN (IP Tunnelling)

 

IP tunnelling ca be used to forward packets from one subnet ot virtual LAN (VLAN) to another subnet or VLAN even when the packets must pass through another network. Building on the IP Tunnelling capability that is part of the Linux Kernel, the LVS-TUN forwarding method. The LVS-TUN forwarding method allows you to place the cluster nodes on a cluster network that is not on the same network segment as the Director.

The LVS-TUN enhances the capability of the LVS-DR method of packet forwrding by encapsulating inbound requests for cluster service from the client computers so that they can forwarded to cluster nodes that are no on the same physical network segment as the Director.This is done by encapsulating one packet inside another packet.

Basic Properties of LVS-TUN

  1. The cluster nodes do not need to be on the same physical network segment as the Director
  2. The RIP addresses must not be private IP Addresses
  3. Return packet must not go through the Director.
  4. The Director cannot remap network port number
  5. Only Operating Systems that supports the IP tunnelling protocol can be the servers inside the cluster.
  6. The LVS-TUN is less reliable than the LVD-DR as anything that breaks the connection between the Director and the cluster nodes will drop all client connections.

For more information on LVS Scheduling methods, see Linux Virtual Server Scheduling Methods Blog entry