CS140: Parallel Scientific Computing: Winter 2009

Introduction to the theory and practice of parallel processing and high-performance computer architecture. Topics include: parallel algorithm design for scientific computation; message-passing parallel programming using MPI; multicore parallel programming using Cilk++; parallel performance evaluation and tuning; parallel numerical algorithms and applications. Students will gain experience with programming some of the world's fastest machines.

Instructor

• Email: gilbert@cs.ucsb.edu
  
• Office: 5109 Harold Frank Hall
  
• Office hours: see my web page.

Teaching Assistant

• Email: sgk@cs.ucsb.edu
  
• Phone: 805-893-7520 (email preferred)
  
• Office hours: Tuesdays and Thursdays 2:00-3:00 in 2152a Harold Frank Hall.

Discussion and Announcements

There is a Google discussion group for the class at http://groups.google.com/group/ucsb-computer-science-140-winter-2009. All students are expected to join this group and to watch it for course announcements. You may also use it to post questions for the instructor and assistant (and answers to other people's questions). The mailing address for posting to the group is ucsb-computer-science-140-winter-2009@googlegroups.com.

Class Meetings

• Lecture: Tuesday and Thursday 9:30-10:45, North Hall 1111
  
• Discussion: Friday 12:00-12:50, Building 387, room 103 (next to Psych)

Textbook

The textbook is: Parallel Programming: Techniques and Applications Using Networked Workstations and Parallel Computers (second edition, 2005), by Barry Wilkinson and Michael Allen. The book has an online home page with sample programs.

I will assign reading from the textbook each week; see the daily schedule below. You will need to both read the text and come to class. Quite a bit of the material we cover in class will not be in the textbook; you are responsible for material we cover in class in any case. Class participation is expected and will have an influence on your grade.

Computer Resources

• You will use two parallel machines in this course, both at the San Diego Supercomputer Center. You will use the TeraGrid IA-64 cluster for MPI (message-passing) homeworks, and the OnDemand cluster for Cilk (multicore) homeworks. Your accounts on TeraGrid and OnDemand will be created automatically if you were preregistered for CS140 by January 2. If not, you must send me email immediately with your name and perm number. It takes a week or so to set up accounts, so please do this right away. In class on Tuesday, January 13, Stefan Boeriu from ECI will give an overview and tutorial on using TeraGrid.

MPI Resources

You will write message-passing programs in C using MPI, which is a library that is implemented on almost all parallel systems. Here are some good references on MPI. You can find more by Googling around on the web.

• The textbook has an introduction to MPI in Sections 2.1 and 2.2, and its Appendix A describes 19 of the MPI routines. MPI has hundreds of routines in all, but these 19 will be more than you need for all the MPI programming you'll do in CS 140.

• Stefan Boeriu's TeraGrid information page (includes lots of info on MPI performance and trace tools, such as VAMPIR).
• User's Guide to MPI, by Peter Pacheco.
• Documentation for all the MPI routines.
• Tutorial on MPI from Livermore Labs, with lots of examples.
• Some examples of MPI code from class and section.
• Norm Matloff's MPI page at UC Davis has lots of sample programs and many other pointers.

Cilk Resources

You will write shared-memory multicore programs in Cilk++, which is a version of C and C++ that has been extended for parallel programming. There is a lot of information on Cilk++ at the CilkArts website.

• Cilk e-book
• Stefan Boeriu's OnDemand information page.
• See also the course slides from January 27 - February 3.

Grading

• Homework assignments: 35%
• Midterm exams (2): 30%
• Final project: 35%

Homework Policy

There will be a number (probably 4) of homework assignments, and a final project. The goal of the final project is to demonstrate your ability to design and implement a parallel program to solve a problem in scientific computing. For the homework assignments and final project, I encourage you to work in groups of two, though you may work alone if you wish. You will write a report describing your approach to the final project and presenting your results.

It is your responsibility to manage your time so that you can turn in your assignments when they are due. No late homework will be accepted.

If you have questions about grading of homework, talk to the t.a. first. If you are unable to reach an agreement, make an appointment to talk to me. The statute of limitations for regrades is one week -- that is, any requests for regrades must be made no later than one week after the homework (or exam) was returned in class.

Homework Assignments

• Homework 1 (due Tue Jan 13): Problem
• Homework 2 (due Thu Jan 22): Problem, Matlab code
• Homework 3 (extended to Mon Feb 9): Problem, SUMMA slides, SUMMA paper
• Homework 4 (extended to Fri Feb 20): Problem, data (test and real)

Exams

• Midterm 1: Thursday, February 5. [syllabus] Open textbook and notes; no other books or computers; no sharing books.

• Midterm 2: Thursday, March 5. [syllabus, update] Open textbook and notes; no other books or computers; no sharing books.

Final Project

For the final project, you will choose one of the three problems described at the link below, write a project proposal in which you describe in detail what you plan to do, and then do an implementation and experiment following your proposal.

The projects are intentionally left open-ended so that you can explore. I encourage you to discuss your plans with me and/or Stefan K., in office hours or by email, as you decide what to do. The project proposal is intended as a chance for you to get feedback on your plans before you get too far down the road, so please make your proposal as complete as possible even if you're not yet sure what might change.

If you have a parallel computing project of your own that you would like to work on instead of doing one of these three problems, please talk to us about it. It's okay to do a different project provided it has the right level of parallel-implementation difficulty, and provided you write a thorough proposal describing it.

I strongly prefer that you do the term project in teams of two. If you want to work alone, you must talk to me first. I will expect the graduate students in the course to propose somewhat more ambitious projects than the undergraduates. It is okay for a grad student and an undergrad to work together on a team.

• Links to Final Project Problem Descriptions
  • Game of Life: writeup, sample code
  • Parallel Sorting: writeup
  • Conjugate Gradients: writeup, sample code

• Tuesday, February 24: Project teams and proposals due. Turn in a one- or two-page report giving the names of the students on your team, saying which project you are going to do, and then going into some detail about what algorithms you will implement, what experiments you will do, and what results you will present.

• Thursday, March 5: Progress reports due. Turn in a one- or two-page report giving the status of your project, what is going well and what is giving you difficulty, and any changes you want to make to your proposal.

• Wednesday, March 18: Demos. Instead of a final exam, each project team will give a 10-minute demo in CSIL during our final exam period (8:00am-11:00am on March 18). [schedule]

• Thursday, March 19: Final project reports due. (Use "turnin final@cs140" from CSIL.) Turn in a report of 5-10 pages describing in detail the algorithms you used and the experiments you did, and your interpretation of the results. What turned out as you expected? Did anything about your results surprise you? If so, can you explain it? Use both tables and graphs to show your results. Also turn in the code for your project program.

Class Schedule, Links, Slides, and Reading Assignments

• Tue Jan 6: Introduction. [parallel sorting, bone density modeling]
  Reading: Textbook sections 1.1, 1.2, 1.3 (overview of parallel computers).
  (Homework 1 assigned)

• Thu Jan 8: Message-passing and MPI. [slides on parallel machines; MPI slides; examples of MPI code]
  Reading: Textbook sections 2.1, 2.2 (introduction to MPI).

• Tue Jan 13: SDSC orientation, accounts, and performance analysis tools (Stefan Boeriu, guest lecturer). [Stefan B.'s web page]
  Reading: Textbook sections 2.3, 2.4, 2.5 (analyzing and debugging parallel programs), and sections 3.1 and 3.2.3 (Monte Carlo methods).
  (Homework 1 due, Homework 2 assigned)

• Thu Jan 15: Matrix-vector multiplication and data layout. [summary of complexity notation]
  Reading: Cilk e-book sections 1 and 2 (parallelism, work and span laws).

• Tue Jan 20: Parallel complexity measures, cache issues. [slides on work law and span law, matrix multiplication, complexity notation]

• Thu Jan 22: Parallel matrix multiplication. (Aydin Buluc, guest lecturer.) [slides on matrix multiplication]
  (Homework 2 due, Homework 3 assigned)

• Tue Jan 27: Multicore, shared memory, and Cilk++. (Aydin Buluc, guest lecturer.) [slides on introduction to multicore] [ Cilk++ on OnDemand ]

• Thu Jan 29: Divide and conquer examples using Cilk++. (Aydin Buluc, guest lecturer.) [slides on sorting on multicore ]
  Reading for the academically inclined: book chapter on parallel algorithms by Guy Blelloch and Bruce Maggs. Note that they use the word "depth" instead of "span".

• Tue Feb 3: Scientific examples using Cilk++ (Aydin Buluc, guest lecturer.) [slides on matrix-matrix and matrix-vector multiplication ].

• Thu Feb 5: Midterm 1. [syllabus and rules]

  (Homework 3 due 11:59pm Mon Feb 9)

• Tue Feb 10: Spatially local computations: Game of Life, finite element modeling, temperature problem. [Wikipedia on Life; a java Life simulator; slides on bone density modeling]
  Reading: Textbook sections 6.1 - 6.3 (synchronous iteration).
  (Homework 4 assigned)

• Thu Feb 12: (no class, JRG out sick)

• Tue Feb 17: Stencil computations, Jacobi relaxation. [Matlab transcript, data, and Jacobi code; slides on Jacobi]

• Thu Feb 19: More stencil computation, finite elements. [slides on stencil computations and communication cost]
  Read the requirements for the final project (above on this page).

  (Homework 4 due 11:59pm Fri Feb 20)

• Tue Feb 24: Conjugate gradients, sparse matrix-vector multiplication. [slides on meshes, conjugate gradients, matrices, and graphs]
  (Final project teams and proposals due)

• Thu Feb 26: Matrices and graphs, partitioning. [same slides as last time on matrices and graphs]
  Reading: Berry-Hendrickson paper (passed out in class).

• Tue Mar 3: Graph algorithms: maximal independent set. [Matlab code for sequential and parallel maximal independent sets]

• Thu Mar 5: Midterm 2. [syllabus and rules, syllabus update]
  (Final project progress reports due)

• Tue Mar 10: Exotic architectures: massive multithreading, GPU computing. [Cray XMT multithreading diagram and overview slides; slides on IBM Cell]

• Thu Mar 12: Exotic languages: Star-P, mapreduce.

• Wed Mar 18: Final project demos in CSIL, 8:00-11:00 am. [schedule]

  (Final project reports due 11:59pm Thu Mar 19)