Information on Running MPI jobs on TSCC

Compile MPI programs
By default, you will be using the Portland Group compiler with mpicc.

mpicc -o test1 test1.c

You may attach an optimization flag.

More details from TSCC on compiling MPI

Running an MPI program in an interactive mode
Command qsub -I allows you to submit interactive jobs, where you are given direct access to a set of resources. More details for interactive jobs on TSCC

Use the command
```
qsub -I -l nodes=1:ppn=16 -l walltime=00:15:00 
```
to start an interactive job on a single node with 16 cores for at most 15 minutes.
Once you run it, it will wait until a node is available, and then it will create an SSH session to that node. Your job will last until you exit that SSH session.
Then run an MPI job as:
```
 mpirun -v -machinefile $PBS_NODEFILE  -np 16 ProgramName 
```
This command executes the MPI program with 16 processes on the allocated machines. $PBS_NODEFILE is a shell variable and its value is a file which contains machine names allocated by "qsub -I".
Notes:
- The interactive mode is the best for compiling code and debugging your program in a small setting. To report the execution time of your program, it is better to use the batch mode and the system can allocate more dedicated resources.
- Remember to exit so that you don't waste a bunch of time doing nothing.

Running an MPI program in a batch mode
Job submission is done with a job script file with a number of options. These include the queue, the number of nodes, the title of your job, the maximum time it can take, where to put console output, and email notifications at job start and end, and Linux commands that run your programs.

Make sure your code does not wait for an input from a user keyboard!

The head part of this script file should contain information on the number of machines and cores to be be requested. You always use "hotel" queue to submit your job. More details on how to specify the head part
To start an MPI job after the resource specification, your script can include the two lines in the following format:
```
cd ~/path-to-program
mpirun -v -machinefile $PBS_NODEFILE -np somenumber ProgramName
```
Command
```
qsub job-script-file 
```
will submit your job to the scheduler and assign it a job number. Be careful when you are submitting multiple jobs with the same output file names since they may get overwritten by subsequent or concurrent jobs.
qstat -a and showq -u $USER are useful for examining the current state of your jobs. In qstat, Q means queued, R means running, and C means complete. showq will give you slightly more detailed information about your jobs.
You may delete a job with "qdel jobnumber" where jobnumber was assigned by qsub.

Sample code and makefile at TSCC
You will be able to access the following files at the TSCC login node.

	~tyang/mpiex    ------------ MPI sample code directory (PI calculation, hello program)
	~tyang/mpiex/Makefile    ------------ the makefile for compiling and running a job 
	~tyang/mpiex/run-hello-mpi.job    ------------ an example shell script to be used for job submission