If you are running Chelsio Cards, you may want to specify the mpirun parameters to ensure the
/usr/mpi/intel/openmpi-1.4.3/bin/mpirun -mca btl openib,sm,self --bind-to-core --report-bindings -np $NCPUS -machinefile $PBS_NODEFILE $PBS_O_WORKDIR/$file
–bind-to-core: Bind each MPI process to a core
–mca btl openib,sm,self: (Infiniband, shared memory, the loopback)
For information on Interprocess communication with shared memory,