The Detailed Information can be found at Displaying MPI Debug Information
The I_MPI_DEBUG environment variable provides a convenient way to get detailed information about an MPI application at runtime. You can set the variable value from 0 (the default value) to 1000. The higher the value, the more debug information you get.
High values of I_MPI_DEBUG can output a lot of information and significantly reduce performance of your application. A value of I_MPI_DEBUG=5 is generally a good starting point, which provides sufficient information to find common errors.
Displaying MPI Debug Information
To redirect the debug information output from stdout to stderr or a text file, use the I_MPI_DEBUG_OUTPUT environment variable
$ mpirun -genv I_MPI_DEBUG=5 -genv I_MPI_DEBUG_OUTPUT=debug_output.txt -n 32 ./mpi_program
I_MPI_DEBUG Arguments
| <level> | Indicate the level of debug information provided. |
| 0 | Output no debugging information. This is the default value. |
| 1 | Output libfabric* version and provider. |
| 2 | Output information about the tuning file used. |
| 3 | Output effective MPI rank, pid and node mapping table. |
| 4 | Output process pinning information. |
| 5 | Output environment variables specific to the Intel® MPI Library. |
| > 5 | Add extra levels of debug information. |
| <flags> | Comma-separated list of debug flags |
| pid | Show process id for each debug message. |
| tid | Show thread id for each debug message for multithreaded library. |
| time | Show time for each debug message. |
| datetime | Show time and date for each debug message. |
| host | Show host name for each debug message. |
| level | Show level for each debug message. |
| scope | Show scope for each debug message. |
| line | Show source line number for each debug message. |
| file | Show source file name for each debug message. |
| nofunc | Do not show routine name. |
| norank | Do not show rank. |
| nousrwarn | Suppress warnings for improper use case (for example, incompatible combination of controls). |
| flock | Synchronize debug output from different process or threads. |
| nobuf | Do not use buffered I/O for debug output. |
References: