Tapestry Logo

Simbmark documentation

Jeremy Stribling
July 2, 2003  

 

 

Programs/packages needed to run simbmark:
The simbmark (short for simulation benchmark) script is a great way to test structured peer to peer networks in an environment that's controlled, reproducible, and which allows for hundreds or thousands of nodes to be simulated on a single machine. This environment is called the Simple OceanStore Simulator (SOSS), the source code for which can be found in the current release of Tapestry under src/soss.

The benchmarks are used to test the reliability and performance characteristics of P2P overlays that are SOSS- and Tapestry-compatible. Currently, there are two types of benchmarks: find owner and object location.

In the "find owner" benchmark, each node picks a random ID in the name space and routes a message to the owner of that ID and gets a response back. The test measures the percentage of successful queries (that is, queries that did not time out and for which some response was received) as well as the percentage of consistent responses. To measure consistency, we coordinate groups of nodes that search for the random IDs at the same time, and see how many of them get the same answer back for the owner of the ID. Latency of queries and bandwidth used is also measured.

The "object location" benchmark consists of every node publishing some number of objects at the beginning, and then having all nodes try routing to random objects that have been published. Here only the percentage successful is measured; that is, the queries that were routed to the node that actually did publish the object. Again, latency and bandwidth are measured as well.

To make things interesting, these tests can be performed under various network-stressing conditions. In one scenario, there can be a massive failure and a massive joining of nodes into the network (at separate times). In another, nodes can "churn" continuously, joining and leaving the network at some specified rate. These network conditions demonstrate how the P2P overlay will maintain its performance under realistically (albeit controlled) dynamic membership. When nodes fail, they fail completely (i.e. failstop).

When, the test is finished, the script will automatically produce a graph of the results and display it on the screen for you.

To see how to run the test, and the different parameters it accepts, just run './run-simbmark --help' from the pond/run/simbmarks directory. Basically, you can specify the size of the P2P overlay network, the amount of (simulated) time for which the test will run, and various parameters for the network events. You can also control certain aspects of the graph that will be produced, and whether or not specific files should be saved after the test is over. If you want all files to be saved after the test is over, set the environment variable OSTORE_DISABLE_CLEANUP to be "true" before running the script.

During the test, the output of the experiment will be saved in your temp directory (/tmp by default, can be specified through the OSTORE_TMP environment variable), as experiment-<PID>-soss.log. There will also be lots of other fun files in this directory (configuration files mostly) while the test is running; they'll be deleted after the test unless the aforementioned environment variable is set.