gpu

by

Issues installing using RHEL8 repo

The Forum found here helped with me RHEL8 repo. I was using Rocky Linux 8.10

dnf install cuda
Last metadata expiration check: 2:34:35 ago on Tue 14 Jan 2025 08:28:15 AM +08.
Error: 
 Problem: package cuda-12.6.3-1.x86_64 from cuda-rhel8-x86_64 requires nvidia-open >= 560.35.05, but none of the providers can be installed
  - cannot install the best candidate for the job
  - package nvidia-open-3:560.28.03-1.noarch from cuda-rhel8-x86_64 is filtered out by modular filtering
  - package nvidia-open-3:560.35.03-1.noarch from cuda-rhel8-x86_64 is filtered out by modular filtering
  - package nvidia-open-3:560.35.05-1.el8.noarch from cuda-rhel8-x86_64 is filtered out by modular filtering
  - package nvidia-open-3:565.57.01-1.el8.noarch from cuda-rhel8-x86_64 is filtered out by modular filtering
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

In the forum, there was a workaround solution

Go to the download page and select the V100 driver.

dnf install. ./nvidia..........x86_64.rpm

Remove the old cuda if you have it installed and reset the repo module streams.

dnf remove cuda-toolkit nvidia-driver-cuda
dnf module reset nvidia-driver

Install dkms and cuda

dnf module install nvidia-driver:latest-dkms 
dnf install cuda-toolkit nvidia-driver-cuda

Alternatively, this method works especially if you have not manually install the drivers or manually uninstall it already.

dnf install cuda-toolkit nvidia-driver-cuda
by

Nvidia-smi slow startup fix

If you encounter slow nvidia-smi before the information is shown. For my 8 x A40 Cards, it took about 26 seconds to initialise.

The reason for slow initialization might be due to the driver persistence issue. For more background on the issue, do take a look at Nvidia Driver Persistence. According to the article,

The NVIDIA GPU driver has historically followed Unix design philosophies by only initializing software and hardware state when the user has configured the system to do so. Traditionally, this configuration was done via the X Server and the GPUs were only initialized when the X Server (on behalf of the user) requested that they be enabled. This is very important for the ability to reconfigure the GPUs without a reboot (for example, changing SLI mode or bus settings, especially in the AGP days).

More recently, this has proven to be a problem within compute-only environments, where X is not used and the GPUs are accessed via transient instantiations of the Cuda library. This results in the GPU state being initialized and deinitialized more often than the user truly wants and leads to long load times for each Cuda job, on the order of seconds.

NVIDIA previously provided Persistence Mode to solve this issue. This is a kernel-level solution that can be configured using nvidia-smi. This approach would prevent the kernel module from fully unloading software and hardware state when no user software was using the GPU. However, this approach creates subtle interaction problems with the rest of the system that have made maintenance difficult.

The purpose of the NVIDIA Persistence Daemon is to replace this kernel-level solution with a more robust user-space solution. This enables compute-only environments to more closely resemble the historically typical graphics environments that the NVIDIA GPU driver was designed around.

Nvidia Driver Persistence

The Solution is very easy. Just start and enable nvidia-persistenced

# systemctl enable nvidia-persistenced
# systemctl start nvidia-persistenced

Immediately, the nvidia-smi command becomes more responsive