The Detailed Information can be found at Displaying MPI Debug Information
The I_MPI_DEBUG environment variable provides a convenient way to get detailed information about an MPI application at runtime. You can set the variable value from 0 (the default value) to 1000. The higher the value, the more debug information you get.
High values of I_MPI_DEBUG can output a lot of information and significantly reduce performance of your application. A value of I_MPI_DEBUG=5 is generally a good starting point, which provides sufficient information to find common errors.
Displaying MPI Debug Information
To redirect the debug information output from stdout to stderr or a text file, use the I_MPI_DEBUG_OUTPUT environment variable
$ mpirun -genv I_MPI_DEBUG=5 -genv I_MPI_DEBUG_OUTPUT=debug_output.txt -n 32 ./mpi_program
I_MPI_DEBUG Arguments
<level> | Indicate the level of debug information provided. |
0 | Output no debugging information. This is the default value. |
1 | Output libfabric* version and provider. |
2 | Output information about the tuning file used. |
3 | Output effective MPI rank, pid and node mapping table. |
4 | Output process pinning information. |
5 | Output environment variables specific to the Intel® MPI Library. |
> 5 | Add extra levels of debug information. |
<flags> | Comma-separated list of debug flags |
pid | Show process id for each debug message. |
tid | Show thread id for each debug message for multithreaded library. |
time | Show time for each debug message. |
datetime | Show time and date for each debug message. |
host | Show host name for each debug message. |
level | Show level for each debug message. |
scope | Show scope for each debug message. |
line | Show source line number for each debug message. |
file | Show source file name for each debug message. |
nofunc | Do not show routine name. |
norank | Do not show rank. |
nousrwarn | Suppress warnings for improper use case (for example, incompatible combination of controls). |
flock | Synchronize debug output from different process or threads. |
nobuf | Do not use buffered I/O for debug output. |
References: