CS170 Lab5: WeensyOS Virtual Memory

Lab5: WeensyOS Virtual Memory


This lab will introduce you to another WeensyOS operating system. (Recall that WeensyOS is a series of coding exercises, written by Eddie Kohler. All of this lab, including the description, is due to Eddie.) In this WeensyOS, you will implement process memory isolation, virtual memory, and some system calls.

Getting Started

We recommend doing this lab inside the development platform that you set up in the previous labs (the virtual devbox or the CSIL machines).

From within this platform, change to your lab directory, use Git to commit changes you've made since handing in lab 4 (if any), obtain the latest version of the course repository, and create a local branch called lab5 based on origin/lab5 (which is our lab5 branch). The commands are as follows, but you may not need all of them (for example, if git status indicates that there are no changes to commit):

$ cd ~/cs170
$ git commit -am 'my solution to lab4'
Created commit 254dac5: my solution to lab4
 3 files changed, 31 insertions(+), 6 deletions(-)
$ git pull
Already up-to-date.
$ git checkout -b lab5 origin/lab5
Branch lab5 set up to track remote branch refs/remotes/origin/lab5.
Switched to a new branch "lab5"
$ cd lab5

You are now in the lab5 directory.

Initial State

Run make run. You should see something like this, which shows four versions of the p-allocator process running in parallel:

This image loops forever; in an actual run, the bars will move to the right and stay there. Don't worry if your image has different numbers of K's or otherwise has different details.

If your bars run painfully slowly, edit the p-allocator.c file and reduce the ALLOC_SLOWDOWN constant.

Stop now to read and understand p-allocator.c. Here’s what’s going on in the physical memory display.

Here are two labeled memory diagrams, showing what the characters mean and how memory is arranged.

The virtual memory display is similar.


You will implement complete and correct memory isolation for WeensyOS processes. Then you'll implement full virtual memory, which will improve utilization. You'll implement fork: creating new processes at runtime. Finally, for extra credit, you'll implement exit.

As usual, the code that you write will be limited. Our solutions contain less than 200 lines. All your code goes in kernel.c.


Running WeensyOS

Read the README-OS.md file for information on how to run WeensyOS. If QEMU’s default display causes accessibility problems, you will want to run make run-console. To make run-console the default, run export QEMUCONSOLE=1 in your shell.

For debugging convenience, we have provided two printf functions:

debug_printf(format, ...)
log_printf(format, ...)

debug_printf will print one line of output at the bottom of the animation screen. Each time you invoke this function, it will overwrite the previous output.

log_printf will print the output into a file log.txt

We recommend using make run-gdb for debugging, as well as adding log_printf statements to your code.

Memory system layout

WeensyOS's memory system layout is described by several constants.

KERNEL_START_ADDR Start of kernel code.
KERNEL_STACK_TOP Top of kernel stack. The kernel stack is one page long.
console CGA console memory.
PROC_START_ADDR Start of application code. Applications should not be able to access memory below PROC_START_ADDR, except for the single page at console.
MEMSIZE_PHYSICAL Size of physical memory in bytes. WeensyOS does not support physical addresses ≥ MEMSIZE_PHYSICAL. Equals 0x200000 (2MB).
MEMSIZE_VIRTUAL Size of virtual memory. WeensyOS does not support virtual addresses ≥ MEMSIZE_VIRTUAL. Equals 0x300000 (3MB).

Address composition

WeensyOS uses several macros to handle addresses. They are defined at the top of x86.h. The most important include:

PAGESIZE Size of a memory page. Equals 4096 (or, equivalently, 1 << 12).
PAGENUMBER(addr) The page number for the page containing addr. Expands to something like addr / PAGESIZE.
PAGEADDRESS(pn) The initial address in page number pn. Expands to something like pn * PAGESIZE.
PTE_ADDR(entry) The physical address within a page table entry. Remember that a page table entry contains 32 bits, with the bottom 12 bits being configuration and the top 20 bits identifying the page number. This returns the top 20 bits as an address (by masking out, or clearing, the bottom 12 bits).

Kernel and process address spaces

WeensyOS begins with the kernel and all processes sharing a single address space. This is defined by the kernel_pagetable page table. kernel_pagetable is initialized to the identity mapping: virtual address X maps to physical address X.

As you work through the lab, you will shift processes to using their own independent address spaces, where each process can access only a subset of physical memory.

The kernel, though, still needs the ability to access any location in physical memory. Therefore, all kernel functions run using the kernel_pagetable page table. Thus, in kernel functions, each virtual address maps to the physical address with the same number. The exception function explicitly installs kernel_pagetable when it begins.

WeensyOS system calls are more expensive than they need to be, since every system call switches address spaces twice (once to kernel_pagetable and once back to the process’s page table). Real operating systems avoid this overhead. In real OSes kernels access memory using process page tables, rather than a kernel-specific kernel_pagetable. This makes the kernel code more complicated, since kernels can’t always access all of physical memory directly.

Exercise 1: Kernel isolation

WeensyOS processes could stamp all over the kernel’s memory if they wanted. Better stop that. Change kernel, the kernel initialization function, so that kernel memory is inaccessible to applications—except for the memory holding the CGA console (the single page at (uintptr_t) console == 0xB8000).

When you are done, WeensyOS should look like this. In the virtual map, kernel memory is no longer reverse-video, since the user can’t access it. Note the lonely CGA console memory block.


Exercise 2: Isolated address spaces

Implement process isolation by giving each process its own independent page table. Your OS should look like this:

Thus, each process only has permission to access its own pages. You can tell this because only its own pages are shown in reverse video.

What goes in per-process page tables:

How to implement per-process page tables:

Exercise 3: Virtual page allocation

So far, WeensyOS processes use physical page allocation: the page with physical address X is used to satisfy the sys_page_alloc(X) allocation request for virtual address X. This is inflexible and limits utilization. Change the implementation of the INT_SYS_PAGE_ALLOC system call so that it can use any free physical page to satisfy a sys_page_alloc(X) request.

Your new INT_SYS_PAGE_ALLOC code must perform the following tasks.

Don’t modify the physical_page_alloc helper function, which is also used by the program loader. You can write a new function if you want.

Here’s how our OS looks after this step.


Exercise 4: Overlapping address spaces

Now the processes are isolated, which is awesome. But they’re still not taking full advantage of virtual memory. Isolated address spaces can use the same virtual addresses for different physical memory. There’s no need to keep the four process address spaces disjoint.

In this step, change each process’s stack to start from address 0x300000 == MEMSIZE_VIRTUAL. Now the processes have enough heap room to use up all of physical memory!

If there’s no physical memory available, sys_page_alloc should return an error to the caller (by returning -1). (Our solution additionally prints “Out of physical memory!” to the console when this happens; you don’t need to.)

Exercise 5: Fork

We return in this exercise to a topic that we saw earlier in the semester: the fork() system call. In the first WeensyOS lab, you implemented a primitive version of fork. In this exercise, you will implement a more realistic fork; this one will actually give the new process a separate memory address space.

Backing up a bit, recall that fork is one of Unix’s great ideas. It starts a new process as a copy of an existing process. (fork returns in each process, the original and the copy.) To the child process, it returns 0. To the parent process, it returns the child’s process ID.

Now, run WeensyOS with make run or make run-console. At any time, press the ‘f’ key. This will soft-reboot WeensyOS and ask it to run a single p-fork process, rather than the gang of allocators. You should see something like this:

Your job now is to implement (most of) fork.

When you’re done, you should see something like this after pressing ‘f’.

An image like this means you forgot to copy the data for some pages, so the processes are actually sharing stack and/or data pages:

Handing in the lab

If you have not done so yet, commit your code.

$ git status  # see what files you changed
$ git diff    # examine your changes
$ git commit -am "My solutions to lab5"