MPI related Errors for ANSYS-20.1


One of my users was facing the issues when running ANSYS-20.1 Fluent

Error: eval: unbound variable
Error Object: c-flush
...
...
...
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
fluent_mpi.20.1.0: Rank 0:110: MPI_Init: ibv_create_qp(left ring) failed
fluent_mpi.20.1.0: Rank 0:110: MPI_Init: probably you need to increase pinnable memory in /etc/security/limits.conf
fluent_mpi.20.1.0: Rank 0:110: MPI_Init: ibv_ring_createqp() failed
fluent_mpi.20.1.0: Rank 0:110: MPI_Init: Can't initialize RDMA device
fluent_mpi.20.1.0: Rank 0:110: MPI_Init: Internal Error: Cannot initialize RDMA protocol
...
...
...

There were a few steps that we took to resolve the issues. Do look at the generated logs which will give lots of information. If you are using the PBS-Professional. your fluent script may look like this.

fluent 3ddp -g -t$nprocs -mpi=ibmmpi -cflush -ssh -lsf -cnf=$PBS_NODEFILE -pinfiniband -nmon -i $inputfile >& $PBS_JOBID.log 2>&1
  • Do note that “-cflush” and that it should be placed after the -mpi flag. From our observation, if the -cflush is placed before -mpi, it seems to initiate the flush, but another error would be generated.
  • Make sure you choose the right mpi parameter.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.