Project #3: Implementation Suggestions for Virtual Memory Management and File-System

These suggestions were written by a previous CS170 TA.
They may be useful for some of you and you DO NOT have to read these suggestions.

Part I (Virtual Memory)

Reccomended Implementation Stages

Part II (File System)

Reccomended Implementation Stages

For this part, you will primarily add files in vm directory and revising files in userprog and machine directories. The following design is one possible approach and other designs are also possibles.

Step 1:
Understand how address translation works in method Machine::Translate in machine/translate.cc, where this method is used, and how PageFaultException can be thrown in this process. Figure out how memory access is implemented and where the virtual to physical translation is performed in using the above method. You can trace the MIPS simulator (in machine/mipssim.cc) in executing instructions. As a part of executing an instruction, the machine accesses main memory for loading the instruction itself and possibly for its operands. It is during the translation process that the machine can determine if the virtual address it is trying to access belongs to a page which does not reside in physical memory. Figure out how, when and where the PageFaultException is thrown.

Step 2:
The above Exception is supposed to be handled in a manner similar to system calls. Currently, this exception is not handled. Add code to exception.cc to call a stub routine when this Exception is raised. This will be your page fault handler. As a part of raising the Exception, the processor saves the faulting virtual address in a register that will be used by the kernel to handle the Exception.

Step 3:
Modify Project 2 code and start a process with none of its pages in memory and you may also need to modify the pagetable structure (in the TranslationEntry class) to keep track of pages that are not in memory. Keep track of the location from which disk-resident pages are to be loaded. Once a page has been brought in to memory, any subsequent flush of this page to disk during page replacement should be to a backing store. You will also need to allocate space in the backing store for this process. You can choose to be conservative and allocate space for the entire virtual address space of the process on the backing store at creation time. You can be even more conservative and choose to copy the entire executable file into the allocated space at startup. If you did this, you would need to only concern yourself with moving pages between backing store and the memory during page fault handling.

Step 4:
Implement a page replacement algorithm. Demand-paged virtual memory makes it possible to run a process with only a fraction of its address space resident in physical memory. Demand-paged virtual memory is supported by address translation hardware that checks each access to main memory to see whether the referenced page is currently resident in physical main memory. If the page is resident, the access proceeds in the normal way. If the page is not resident, then a page fault is generated, and the CPU leaves user mode and begins executing a kernel routine whose purpose is to handle such exceptions. This routine, called the page fault handler, must perform the following tasks:

• If necessary, allocate space on the backing store to receive the contents of the victim page (assuming that it is dirty and needs flushing).
• Initiate I/O to write the contents of the victim page to the backing store.
• Adjust the pagetable for the process to which the victim page belongs to reflect the fact that it is no longer resident in memory.
• Locate the page for which the fault was generated on the backing store; initiate I/O to load the page into the page frame selected in the previous steps.
• Adjust the pagetable for the faulting process to reflect the fact that the desired page is now resident in memory.
• Return to user mode and restart the instruction that caused the fault. Make sure to re-execute the instruction for which the page-fault was generated in the first place.

1. A swap map is needed to keep track of the allocation of space on the backing store. This can be a bitmap with one bit for each sector of backing store.
2. A table that has one entry for each page of physical memory, giving information about that page, such as the last time it was referenced and and a pointer to the Thread or AddrSpace which has data resident in that page.
3. Additional field in the TranslationEntry class that makes up an AddrSpace's page table to represent wether or not the page is on disk or in Memory.

Once you get this working, you should be able to execute programs normally. Launch multiple processes in Nachos simultaneously (using exec, fork) and test your code under various conditions of system load. Include a test case using one process with an address space larger than physical memory and a test case using several concurrently running processes with combined address spaces larger than physical memory. The sort program in the test directory is an example of a program designed to stress the virtual memory system.

You should disable the TLB feature available in Nachos by removing a TLB flag from the makefile in the vm directory.

Implementation Guide for Part II

Step 1: Understand how the Nachos filesystem operates and how Nachos implements files.
Review the Create and Open functions in filesys.cc and you will later modify the Create and Open system calls to use these functions.
Review openfile.cc. Examine the FileHeader class and see why the file size is limited.

Step 2: Modify the file system calls from Project 2 to use Nachos file system. These file system calls (Create, Open, Read, Writei, and Close) have invoked the file system stubs located in filesys.h defined under FILES_STUB. The Nachos that is built under the filesys/ directory uses the calls defined in filesys.h under the #defines for FILESYS.
Make the necessary modifications to your system calls to allow them to use the FILESYS version of these calls.

Step 3: Modify Nachos file system so that you can create larger files. The full disk size is defined disk.h in machine directory. You would need to implement indirect blocks in order to keep track of sectors used for large files. Make sure your file header is not larger than one disk sector. It is not acceptable to allocate all the sector pointers in the FileHeader all at once. In other words, each header should only have the capcity to handle the current file size and will need to be allocate more single indirection blocks when the file is extended to a size that requires the exta header block pointers.

Step 3a: Modify the FileHeader object to contain a sector number that points to a block of pointers (single indirect). The single indirect block can be implemented as a new class called IndirectPointerBlock which will take up an entire sector (just like FileHeader), but contain nothing but pointers to sectors. The IndirectPointerBlock should support the following methods:
1. WriteBack(int sector)
   Just like WriteBack in FileHeader, this will write the object (this *) to the sector of the disk passed in as an argument.
2. FetchFrom(int sector)
   Just like FetchFrom in FileHeader, this method will de-serialize the object that was written to the sector argument.
3. PutSector(int sector)
   This method will add a newly allocated sector to the block of block pointers. The FileHeader will perform all the allocations and pass the sector numbers to the IndirectPointerBlock via this method. This will be easier than having each pointer block manage sectors with a single free map.
4. Deallocate(BitMap *freeMap)
   Just like Deallocate in FileHeader, this method will loop through the sectors allocated and un-mark them from the free map to signify thier deletion.
5. ByteToSector(int sector)
   This method will return the sector number for the byte offset relative to the beginning of its IndirectPointerBlock. So if an IndirectPointerBlock covers blocks 60-90 and I pass in an offset of SectorSize*10, I know that the sector number can be found in the 10th entry.

Modify the FileHeader's Deallocate and Allocate methods to create this single indirect pointer block when necessary and deallocate it when it's no longer needed. You will only deallocate once when the whole file is destroyed, so it is OK to just call Deallocate recursivly Modify the FileHeader's FetchFrom and WriteBack methods to recursivly put and get the IndirectPointerBlock. Modify the ByteToSector method to query the single indirect pointer block for the location of a byte using the IndirectPointerBlock's ByteToSector method.
Now you may have 5 direct block pointers and 25 single IndirectPointerBlocks in the file header, which allows that the maximum file size exceeds 100K bytes.

Step 4: Allow the Write system call to extend the size of a file. Currently, if the user tries to write beyond the end of the file,Nachos returns an error. Modify the current implementation so that a write beyond the end of the file extends the size of the file. The exception is that if the file is created to be a given size, your implementation should allocate as many sectors (and block pointers) as required to hold that amount of data. For now this function can allocate all pages that are needed in order to fill the gap between the previous end sector and the new end sector. Gracefully handle the case of the disk being full - the user process should not crash. Instead an error should be returned.

Step 4a: Leave the ReadAt() method alone. If a user tries to read beyond the end of a file they should get back nothing. If they start the read in a valid position, but try to read too much, they should only get back the amount that existed between the starting position and the end of the file. That's the way the function works now and that's the way it should work.

Step 4b: Modify the WriteAt() method in openfile.cc to check only for writes that go beyond the size of the disk (not the current size of the file). Modify the reading of the first and last sector to handle the case when you are writing beyond the current bounds of the file. I would switch() the cases that can occur as such:
• Case: Write request is within the bounds of the file. Copy the existing code in a spot that handles this case. It's ugly and you don't want to have to generalize it to work for the remaining cases.
• Case: Write request overwrites some portion of existing file and goes beyond by N bytes. You only need to read in the last sector, but will need to extend the file before the buffer can be written back.
• Case: Write request overwrites no data, but goes beyond the end of the file. Here, no data needs to be read in, you just need to extend the file and begin writing at the sectors given to you by ByteToSector() Make sure you don't attempt to extend the file beyond 1024 bytes.


Step 4c: Include a routine in FileHeader that extends the file by N sectors. Name it something like: ExtendFile. Call this file with the proper argument before writing the bytes back to disk (in the WriteAt() function).