Please read this post to avoid excessive usage of the cluster. This content is also included in the tutorials.
Step 1: Before submitting a SLURM job script, check the current GPU node usage. Type ssh gpu_node to log in to the specific GPU node where you want to run your project.
Step 2: Type nvidia-smi to check the current usage of the gpu node.
Step 3: Check the “Volatile GPU-Util” in the prompt screen. If it is not 0%, then someone already occupy their job in that node. Make sure avoid to use the same node to prevent excessive usage.
Step 4: Repeat the same steps after submitting your job. If ‘Volatile GPU-Util’ is close to 100%, the node has reached its full capacity, which could cause potential hardware issues. In this case, reduce the batch size or model size, clear the caches, or wait until other jobs are completed
– RebelX Team