Information on Comet Cluster Usage

Comet Cluster

Comet cluster has 1944 nodes and each node has 24 cores, built on two 12-core Intel Xeon E5-2680v3 2.5 GHz processors with 128 GB memory and 320GB SSD for local scratch space. The cluster also contains 36 nodes with GPUs. Detailed information can be found from XSEDE website.

Here are some questions and answers related to CS240A.

How to login and problems with the login node
To use Comet, log in to the front-end machine: comet.sdsc.xsede.org.
The files you see from the login node are typically shared among all nodes in the cluster. But use this login machine only for compiling and general tasks. For running MPI or other parallel jobs, use the queuing system discussed below to access the computer cluster.
Comet imposes a memory constraint on each user process running on this login node and you may observe strange problems such as unexpected segmentation fault when running a program that requires a modest allocation of memory (through object creation, malloc, or even regular function calls). You should run code through sbatch. If you really want an interactive debugging or do some memory-intensive compilation (such as Java), then allocate a dedicated node for interactive debugging (see below).
How to check account balance?
Every class user is allocated with a certain amount of service units. Use the show_accounts command to check the amount of time in your account.
Your account is normally billed according based on the nearest CPU node hour. Comet has its specific policy on charging and please read Comet web document for details.
It is easy to accidentally chew through your time allocation by running a program with dead lock or leaving an interactive session open. One thing you can do is to always specify a maximum time the job can execute. For example, 10-20 minutes.
How to compile and submit an openmp/MPI job?
Here are some sample code and Makefile instructions accessible from your Comet account on how to submit a parallel job using sbatch. Command "sbatch" allocates a set of nodes to execute a parallel job.
- ```
/home/tyang/cs240sample/mpiex 
```
  Sample MPI code. The makefile has an entry on how to submit MPI hello job. run-mpi-comet is the sample script used in this sample submission. Command "ibrun" in the script is used to execute an MPI job once a set of nodes are allocated via sbatch.
- ```
/home/tyang/cs240sample/openmpex 
```
  Sample OpenMP code. The makefile has an entry on how to submit a hello openMP job with sbatch. The script run-openmp-comet is the sample script invoked by using sbatch.
How to check if your job is running and cancel a job if needed?
Use the squeue -u command to check the job status.
Use the scancel command to cancel a job.
How to allocate a single node to execute some compute-intensive job interactively?
Use the following command which will allocate such a node for at most 5 minutes.
/share/apps/compute/interactive/qsubi.bash -p compute --nodes=1 --ntasks-per-node=1 -t 00:05:00 --export=ALL
For example, you can use such a command to allocate one node and compile a java program. Comet does not let you compile a java program in the login node.
DONOT FORGET to exit this interactive job. To do that, just type "exit".