CS140: Parallel Scientific Computing: Winter 2011

Introduction to the theory and practice of parallel processing and high-performance computer architecture. Topics include: parallel algorithm design for scientific computation; message-passing parallel programming using MPI; multicore parallel programming using Cilk; parallel performance evaluation and tuning; parallel numerical algorithms and applications. Students will gain experience with programming some of the world's fastest machines.

Instructor

• Email: gilbert@cs.ucsb.edu
  
• Office: 5109 Harold Frank Hall
  
• Office hours: see my web page.

Teaching Assistant

• TA Page: Computer Resources, MPI Resources, Discussion Section Notes

Engineering College Supercomputing Consultant

• Email: stefan@engineering.ucsb.edu
  
• Office: Harold Frank Hall 3116
  
• Phone: 893-8287
  
• Stefan's web page on Triton stuff

Discussion and Announcements

There is a Google discussion group for the class at http://groups.google.com/group/ucsb-computer-science-140-winter-2011. All students are expected to join this group and to watch it for course announcements. You may also use it to post questions for the instructor and assistant (and answers to other people's questions). The mailing address for posting to the group is ucsb-computer-science-140-winter-2011@googlegroups.com.

Class Meetings

• Lecture: Monday and Wednesday 9:30-10:45, North Hall 1111
  
• Discussion: Friday 11:00-11:50, Building 387, room 104 (next to Psych)

Textbook

The textbook is: Parallel Programming: Techniques and Applications Using Networked Workstations and Parallel Computers (second edition, 2005), by Barry Wilkinson and Michael Allen. I will assign reading from the textbook each week; see the daily schedule below. You will need to both read the text and come to class. Quite a bit of the material we cover in class will not be in the textbook; you are responsible for material we cover in class in any case. Class participation is expected and will have an influence on your grade.

Computer Resources

• You must generate a public key and send it to Stefan no later than Wednesday, January 5 in order to get your Triton account in time for the first programming assignment the following Monday.
• See instructions here for how to connect to Triton using your generated key. In class on Monday, January 10, Stefan Boeriu from ECI will give an overview and tutorial on using Triton.
• Triton is the big parallel machine you'll use in this course, located at the San Diego Supercomputer Center. You will do programming homework using both MPI (message-passing) and Cilk (multicore). You can debug Cilk code on the dual-core machines at CSIL or on your own machine if it has multiple cores; you'll do Cilk runs on larger numbers of cores on Triton to turn in. For MPI, you will have to use Triton for debugging unless you have access to another MPI cluster.
• Stefan's web page on Triton stuff

MPI Resources

You will write message-passing programs in C using MPI, which is a library that is implemented on almost all parallel systems. Here are some good references on MPI. You can find more by Googling around on the web.

• The textbook has an introduction to MPI in Sections 2.1 and 2.2, and its Appendix A describes 19 of the MPI routines. MPI has hundreds of routines in all, but these 19 will be more than you need for all the MPI programming you'll do in CS 140.

• User's Guide to MPI, by Peter Pacheco.
• Documentation for all the MPI routines.
• Tutorial on MPI from Livermore Labs, with lots of examples.
• Some examples of MPI code from class and section.
• Norm Matloff's MPI page at UC Davis has lots of sample programs and many other pointers.

Cilk Resources

You will write shared-memory multicore programs in Cilk, which is a version of C and C++ that has been extended for parallel programming. We will use the free version, which is called Cilk++. This version runs on Triton and on the CSIL machines, and you can download it to run on your own multicore laptop or desktop here.

• Cilk e-book
• Download the Cilk++ Programmer's Guide from the Intel webpage
• Multithreaded Algorithms chapter of the CLSR book (Introduction to Algorithms).
• Here is a lot of documentation on the non-free version of Cilk, which is called Intel Cilk Plus. There are some minor differences, but most of the advice here is useful: Intel Cilk Plus documentation and resources

Grading

• Homework assignments: 35%
• Midterm exams (2): 30%
• Final project: 35%

Homework Policy

There will be a number (probably 5) of homework assignments, and a final project. The goal of the final project is to demonstrate your ability to design and implement a parallel program to solve a problem in scientific computing. For the homework assignments and final project, I encourage you to work in groups of two, though you may work alone if you wish. You will write a report describing your approach to the final project and presenting your results.

It is your responsibility to manage your time so that you can turn in your assignments when they are due. No late homework will be accepted.

If you have questions about grading of homework, talk to the t.a. first. If you are unable to reach an agreement, make an appointment to talk to me. The statute of limitations for regrades is one week -- that is, any requests for regrades must be made no later than one week after the homework (or exam) was returned in class.

Homework Assignments

• Homework 1 (due Mon Jan 10): Problem
• Homework 2 (due Wed Jan 19): Problem, Matlab code, Input/output spec
• Homework 3 (due Mon Jan 31): Problem, Matlab code, Input/output spec
• Homework 4 (due Mon Feb 7): Problem
• Homework 5 (due Fri Feb 18): Problem, Matlab code, Java applet.

Exams

• Midterm 1: Wednesday, February 2. [syllabus, sample questions] Open book and notes; no electronic devices of any kind; no sharing notes or books.
• Midterm 2: Monday, March 7. [syllabus, sample questions] Open book and notes; no electronic devices of any kind; no sharing notes or books.

Final Project

For the final project, you will choose one of the three problems described below, write a project proposal in which you describe in detail what you plan to do, and then do an implementation and experiment following your proposal.

The projects are intentionally left open-ended so that you can explore. I encourage you to discuss your plans with me, in office hours or by email, as you decide what to do. The project proposal is your chance to get feedback before you get too far down the road, so please make the proposal as complete as possible. If we see problems with your plans, or if your proposal is too vague, we will ask you to redo the proposal until we're comfortable with it.

If you have a parallel computing project of your own that you would like to work on instead of doing one of these three problems, please talk to me about it. It's fine to do a different project provided it has the right level of parallel-implementation difficulty, and provided you write a thorough proposal describing it.

I strongly prefer that you do the term project in teams of two. If you want to make other arrangements, you must talk to me first. I will expect the graduate students in the course to propose somewhat more ambitious projects than the undergraduates. It is okay for a grad student and an undergrad to work together on a team.

• Links to Final Project Problem Descriptions
  • The Graph500 Benchmark: Breadth-First Search (Cilk++) [writeup, example, sequential code and sample graphs]
  • Conjugate Gradients (MPI) [writeup, sample code]
  • All-Pairs Shortest Paths (CUDA) [writeup, sample code]

• Wednesday, February 23: Project teams and proposals due. Turn in a two-page report giving the names of your team members, saying which project you are going to do, and then going into some detail about what algorithms you will implement, what experiments you will do, and what results you will present. (Use "turnin proposal@cs140" from CSIL.)

• Wednesday, March 9: Progress reports due. Turn in a two-page report giving the status of your project, what is going well and what is giving you difficulty, and any changes you want to make to your proposal. (Use "turnin progress@cs140" from CSIL.)

• Thursday, March 17: Demos. Instead of a final exam, each project team will give a 10-minute demo in CSIL between 8:00am and 11:00am on March 17. [schedule]

• Friday, March 18: Final project reports due. Turn in a report of about 10 pages describing in detail the algorithms you used and the experiments you did, and your interpretation of the results. What turned out as you expected? Did anything about your results surprise you? If so, can you explain it? Use both tables and graphs to show your results. Also turn in the code for your project program. (Use "turnin final@cs140" from CSIL.)

Class Schedule, Links, Slides, and Reading Assignments

• Mon Jan 3: Introduction. [parallel machines, Top500 list, Graph500 list, parallel sorting, bone density modeling]
  Reading: Textbook sections 1.1, 1.2, 1.3 (overview of parallel computers).
  (Homework 1 assigned)

• Wed Jan 5: Message-passing and MPI. [MPI slides; examples of MPI code]
  Reading: Textbook sections 2.1, 2.2 (introduction to MPI).
  (Public key due to Stefan)

• Mon Jan 10: SDSC orientation, accounts, and performance analysis tools (Stefan Boeriu, guest lecturer). [Stefan's web page]
  (Homework 1 due, Homework 2 assigned)

• Wed Jan 12: Matrix-vector multiplication, data layout, parallel complexity measures. [slides on complexity notation]
  Reading: Textbook sections 2.3, 2.4, 2.5 (analyzing and debugging parallel programs), and sections 3.1 and 3.2.3 (Monte Carlo methods).

• Mon Jan 17: (Martin Luther King holiday, no class)

• Wed Jan 19: Parallel matrix multiplication. [slides]
  Reading: Cilk e-book sections 1 and 2 (parallelism, work and span laws).
  (Homework 2 due, Homework 3 assigned)

• Mon Jan 24: Models of communication cost for matrix multiplication. [same slides as last time]

• Wed Jan 26: Data locality and cache issues; scientific libraries. [same slides as last time]

• Mon Jan 31: Multicore, shared memory, and Cilk. [slides]
  (Homework 3 due, Homework 4 assigned)

• Wed Feb 2: Midterm 1.

• Mon Feb 7: More Cilk: Sorting, divide-and-conquer algorithms and recurrences. [slides]
  (Homework 4 due, Homework 5 assigned)

• Wed Feb 9: More Cilk: Matrix algorithms. [slides]

• Mon Feb 14: Final projects; spatially local computations: game of Life, temperature problem; stencil computations and Jacobi relaxation. [slides, demos]
  Reading: Final project requirements (above on this page); Textbook sections 6.1 - 6.3 (synchronous iteration).

• Wed Feb 16: Stencils, meshes, and sparse matrices. [slides]

  (Homework 5 due Fri Feb 18)

• Mon Feb 21: (Presidents' Day holiday, no class.)

• Wed Feb 23: Matrices and graphs; conjugate gradients, matrix-vector multiplication. [same slides as last time]
  Reading: Graph analysis with high-performance computing (Hendrickson and Berry).
  (Final project teams and proposals due)

• Mon Feb 28: Graph partitioning; non-mesh graphs from social networks, etc. [updated slides from last time]

• Wed Mar 2: Maximal independent set; graph algorithms and multithreaded architectures; manycore trends, GPUs, and the future. [slides and Matlab code for independent sets]

• Mon Mar 7: Midterm 2.

• Wed Mar 9: TBD.
  (Final project progress reports due)

• Thu Mar 17: Final project demos in CSIL, 8:00-11:00 am. [schedule]

  (Final project reports due 11:59pm Fri Mar 18)