Step 1: Proceed to the Head Node (Scheduler)
Once you have the Job ID you wish to investigate, go to the Head Node and do. The “-n” is the number of days to search logs at
% tracejob -n 10 jobID
From the tracejob, you will be able to take a peek which node the job landed. Next you can go the node in question and find information from the mom_logs
% vim /var/spool/pbs/mom_logs/thedateyouarelookingat
% vim /var/spool/pbs/mom_logs/20201211
Using Vim, search for the Job ID
You should be able to get a good hint of what has happened. In my case is that my nvidia drivers are having issues.