There is an excellent article on how to build OpenMPI Libraries for 64-bit integers. For more detailed information, do look at How to build MPI libraries for 64-bit integers

The information on this website is taken from the above site.

Step 1: Check the integer size. Do the following. If the output is as below, you have to compile OpenMPI with 64 bits.

# ompi_info -a | grep 'Fort integer size'
Fort integer size: 4

Intel Compilers Step 2a: To compile OpenMPI with Intel Compilers and with 64-bits integers, do the following:

# ./configure --prefix=/usr/local/openmpi CXX=icpc CC=icc 
F77=ifort FC=ifort FFLAGS=-i8 FCFLAGS=-i8
# make -j 8
# make install

* GNU Compilers

Step 2b: To compile OpenMPI with GNU Compilers and with 64-bits, do the followings:

# ./configure --prefix=/usr/local/openmpi CXX=g++ CC=gcc F77=gfortran FC=gfortran \
FFLAGS="-m64 -fdefault-integer-8"       \
CFLAGS="-m64 -fdefault-integer-8"      \
CFLAGS=-m64                             \
CXXFLAGS=-m64
# make -j 8
# make install

Step 3: Update your PATH and LD_LIBRARY_PATH in your .bashrc

export $PATH=/usr/local/openmpi/bin:$PATH 
export $LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH

Verify that the installation is correct

# ompi_info -a | grep 'Fort integer size' 
Fort integer size: 8

Taken from OpenMPI FAQ  21. Can I oversubscribe nodes (run more processes than processors)?

Open MPI basically runs its message passing progression engine in two modes: aggressive and degraded.

• Degraded: When Open MPI thinks that it is in an oversubscribed mode (i.e., more processes are running than there are processors available), MPI processes will automatically run in degraded mode and frequently yield the processor to its peers, thereby allowing all processes to make progress.
• Aggressive: When Open MPI thinks that it is in an exactly- or under-subscribed mode (i.e., the number of running processes is equal to or less than the number of available processors), MPI processes will automatically run in aggressive mode, meaning that they will never voluntarily give up the processor to other processes. With some network transports, this means that Open MPI will spin in tight loops attempting to make message passing progress, effectively causing other processes to not get any CPU cycles (and therefore never make any progress).

Example of Degraded Modes (Running 4 Slots on 1 Physical  cores). MPI knows that there is only 1 slot and 4 MPI process are running on the single slot.

$ cat my-hostfile
localhost slots=1
$ mpirun -np 4 --hostfile my-hostfile a.out

Example of Aggressive Modes (Running 4 slots on 4 or more Physical Cores). MPI knows that there is at least 4 slots for the 4 MPI process.

$ cat my-hostfile
localhost slots=4
$ mpirun -np 4 --hostfile my-hostfile a.out

Taken from 17. How do I tell Open MPI to use processor and/or memory affinity in Open MPI v1.4.x? (How do I use the –by* and –bind-to-* options?)

During the mpirun, you can put in the parameter of the Open MPI 1.4 and above to improve performance

1. –bind-to-none: Do not bind processes (Default)
2. –bind-to-core: Bind each MPI process to a core
3. –bind-to-socket: Bind each MPI process to a processor socket
4. –report bindings: Report how the launches processes are bound by Open MPI

If the hardware has multiple hardware threads like those belonging to Hyperthreading, only the first thread of each core is used with the -bind-to-*. According to the article, it is supposed to be fixed in v1.5

The following options below is to be used with –bind-to-*

1. –byslot: Alias for –bycore
2. –bycore: When laying out processes, put sequential MPI processes on adjacent processor cores. (Default)
3. –bysocket: When laying out processes, put sequential MPI processes on adjacent processor sockets.
4. –bynode: When laying out processes, put sequential MPI processes on adjacent nodes.

Finally you can use the –cpus-per-procs which binds ncpus OS processor IDS to each MPI process. If there is a machine with 4 cores and 4 cores, hence 16 cores in total.

$ mpirun -np 8 --cupus-per-proc 2 my_mpi_process

The command will bind each MPI process to ncpus=2 cores. All cores on the machine will be used.