Expanse Cluster

Here are some questions and answers related to CS293S.

• How to get an account an Expanse/ACCESS account?
  Visit ACCESS portal and navigate to https://identity.access-ci.org/new-user . Once you have got your ACCESS account, I need to add your account name and allow it access our shared CPU resource allocation and it may take 24 hours to propagate this account allocation information. Hopefully your username for Expanse may be the same (if you experence a difficulty, please ask in Piazza to see anybody knows a solution). Your password will be the same.

  If you would like to be able to login to the expanse server without using a password, use the guide here

• How to login and problems with the login node
  To use Expanse, log in to the front-end machine: login.expanse.sdsc.edu
  

  $ ssh <username>@login.expanse.sdsc.edu
  

  The files you see from the login node are typically shared among all nodes in the cluster.

  The Expanse login node imposes a memory constraint on each user process so that you cannot run memory-intensive jobs. To run a parallel job or some of sequential jobs, you should submit a job to execute the binary by running through sbatch. If you want an interactive debugging or do some memory-intensive compilation, then allocate a CPU or GPU node for interactive compilation or debugging (see below).

  The login node may not have certain software modules pre-loaded for compiling and other special commands including job submission. You may have to use module load to load some required modules. See more explanations in Expanse guide on compiling and running.

• How to check account balance?
  Our class is allocated with a certain amount of service units under account "csb180", and they are shared among all students. Use the following expanse-client command to check the amount of CPU time and GPU time allocated to the entire class.

  module load sdsc
  expanse-client user -p
  expanse-client project csb180 -r expanse_gpu -p

  Your account is billed based on the nearest CPU or GPU node hour. Expanse has its specific policy on charging and please read Expanse web document for details.

  It is easy to accidentally chew through your time allocation by running a program with dead lock or leaving an interactive session open. One thing you can do is to specify a maximum time the job can execute if possible to avoid the excessive usage. Or cancel your job as shown below if you feel something is wrong.

• Where to store your files?

  Your home directory can hold general files. If your program needs to access some files frequently, or these files are very large, you can use directory /expanse/lustre/projects/csb180/xyz under a parallel file system called Lustre, where csb180 is the class allocation account and xyz is your user name.

• How to allocate a single machine node to execute some code interactively?

  The following examples are modified from Expanse website:

  • Use the following command to allocate such a debugging CPU-only node. It allocates one machine node with one core in the "debug" cluster and runs a Shell program for 10 minutes at most. The CPU usage is charged to account "csb180".

    module load slurm
    srun --pty --nodes=1 --ntasks-per-node=1 -p debug -t 00:10:00 -A csb180 /bin/bash

    DONOT FORGET to exit this interactive job (otherwise your account is still being charged for CPU and/or GPU time). To do that, just type shell command "exit".

  • This allocates a debugging GPU node.

    srun --pty --nodes=1 --ntasks-per-node=1 -p gpu-debug --gpus=1 -t 00:10:00 -A csb180 /bin/bash

    Exit this interactive job with "exit" shell command.

  • An example of running a Python program. Click here for two detailed examples for data classification and document search in Python.

    srun --pty --nodes=1 --ntasks-per-node=1 -p gpu-shared --gpus=1 -t 00:05:00 -A csb180 python example.py

  Notice the cluster partition used is "debug" and "gpu-debug". There are other partitions used for long job submissions. The Expanse webpage has a table under Section "Running jobs on Expanse" that lists all partitions and their restriction.

  If Partition "debug" is not available, use "shared". Please only use "debug", "shared" for CPU-only jobs, and use "gpu-debug" and "gpu-shared" for GPU jobs.

  Do NOT use other partitions which are expensive. For example, you will be charged using the entire cluster with all cores even you only one core/one machine in Partition "compute" or "ind-compute".

• How to submit a long job executed in dynamically allocated machine nodes in a non-interactive mode?

  Submitting a job allows a dynamic allocation of some machine nodes when available. That is required if this job takes time to run.

  Directory /home/tyang/cs140sample/pthreads in the login node of Expanse has some sample code and Makefile instructions. This example uses a multi-threaded C program with a Makefile. You could run any kind of code including Python and Java. You must change the account to be charge as csb180 before submitting your jobsin the job script.

  • If you use Java, you may not be able to use the login node to compile (you need to allocate a cluster node to compile).
  • If you use gcc or other special packages, you may have to load the software package first. For example,
    module reset
    module load gcc
    make

  In this example, the makefile for a multi-threaded C program shows how to submit a job that executes a binary. For example, it uses script run-expanse to submit a job with command "sbatch" in order to execute the binary code named "ex1". Once a job is submitted using "sbatch run-expanse", the system shows a job number and places this job in a queue, waiting for an allocation of required computing nodes using an account (change it as csb180 allocated for this CS293S class). After a set of nodes is allocated to execute this job, the system saves the output of this job in a file specified in the script.

• How to check if your job is running and cancel a job if needed?

  Use the follow command to check the job status:
  module load slurm
  squeue -u UserName

  Use the following command to cancel a job:
  scancel JobID