Programmer's Guide

Tutorial on Tapestry functionality
and programming interface

This document will attempt to bring the reader up to speed on programming applications on top of Tapestry.  I will briefly discuss the event-driven programming model used, the relevant stages required for Tapestry operation, and then discuss in detail the Tapestry interface and messages.

[ Event-driven Model | Tapestry Stages | Tapestry Interface |
Configuration Parameters | Miscellaneous ]

Event-driven Model

The Tapestry and OceanStore prototype code are written in Java, using a event-driven programming model.  Specifically, the Java code uses the asynchronous I/O library from Matt Welsh's SEDA/SandStorm project.  Details on how to use SEDA/SandStorm can be found at the project website.

Briefly, the current Tapestry/OceanStore code structures different threads of operation into Stages which run concurrently inside a single JVM.  Each stage runs as a thread, which starts out by executing some initialization routines and then enters an event loop.  All stages communicate with each other and with external JVMs via messages and events. Inside each JVM, a dispatcher monitors all messages and events, and delivers copies to each stage that subscribes to messages of that type.  For example, a common Tapestry node would include several stages, including a Network stage, a StaticTClient stage, a DynamicTClient stage, a Router stage and a Tapestry application stage.

Specification of which stages run inside a JVM are declared in configuration (.cfg) files.  On startup, SEDA reads the config files, which provide scoped information using an XML-like tag structure.  Given a config file named SampleClient.cfg, you can run the specified JVM by doing:
    run-sandstorm SampleClient.cfg

For example, here is a segment of a configuration file that specifies the Router stage in a Tapestry node:
    <Router>
       class ostore.tapestry.impl.Router
       queueThreshold 1000
       <initargs>
          pkey PublicKey1
          skey PrivateKey1
          dynamic_route static
       </initargs>
    </Router>
This segment specifies that a Router stage should run inside this JVM, names the full classname that provides the event thread (ostore.tapestry.impl.Router), and declares several external arguments.  These arguments can then be retrieved by doing:
    String dynamic_route = config.getString ("dynamic_route");

Note that each Router stage can either use a public and private key pair (generated by running src/ostore/security/genkey) to generate its node ID (default), or it can use a hash of its IP address to generate its node ID (if the config argument fake_keys is set to true).

In each stage's initialization phase, it specifies which event and messages it wants to "listen" to.  Here's some sample code:

String [] msg_types = {
    "ostore.rp.InnerRingCreateReqMsg",
    "ostore.inner.UpdateReqMsg",
    "ostore.inner.LatestHeartbeatReqMsg",
    "ostore.inner.TimedHeartbeatReqMsg",
    "ostore.inner.BlockReadReqMsg"
};

for (int i = 0; i < msg_types.length; ++i) {
    // register the type
    ostore.util.TypeTable.register_type (msg_types [i]);

    // set up a filter
    Filter filter = new Filter ();
    if (! filter.requireType (Class.forName (msg_types [i])))
        BUG (tag + ": could not require type " + msg_types [i]);
    if (! filter.requireValue ("inbound", new Boolean (true)))
        BUG (tag + ": could not require inbound = true for "
                + msg_types [i]);
    _classifier.subscribe (filter, _this_sink);
}
When SandStorm starts up, it reads the configuration file, and allocates threads for each stage.  Each thread runs the initialization code for its stage.  When all threads have finished initializing, SandStorm sends out a StagesInitializedSignal to all stages.  Independent stages wait for this signal to start their execution.  Note that applications dependent on Tapestry must wait for the Tapestry stages to finish before starting.  The signal they must subscribe to and wait for is TapestryReadyMsg.

In the handleEvent loop, write the event handlers for each of the messages you listened for in init. Here's some more sample code:
public void handleEvent(QueueElementIF item)
    throws EventHandlerException {

    if (item instanceof InnerRingCreateReqMsg) {
        InnerRingCreateReqMsg msg = (InnerRingCreateReqMsg) item;
        // do something with it
    }
    // etc.

    else {
        BUG (tag + ": received unknown QueueElementIF: " +item+ ".");
    }
}
To send messages of your own, use ostore.dispatch.Classifier.dispatch (). Like this:
UpdateRespMsg resp_msg = new UpdateRespMsg (signed_resp);
try {
    _classifier.dispatch(item);
} catch (SinkException e) {
    BUG (class_tag + ".dispatch: could not dispatch " + item);
}


Tapestry Stages

To run a Tapestry / OceanStore node, several key Tapestry stages must be included in the configuration file.  There are four required stages: Network, Router, TClient and DTClient.

The Network stage is required by any node that uses network communication.  More importantly, by using the port variable, we can associate a single virtual node of Tapestry / OceanStore with a IP address / Port # combination.  In fact, most OceanStore tests run by executing multiple JVMs on a single physical machine, where each JVM is associated with an IP address -- Port # pair.

The Router stage is the core stage which does the actual message processing of Tapestry.  It accepts inter-node messages from other JVMs, and Tapestry API messages from other stages in the same JVM.  It handles object publish/location and message routing.  Currently, Tapestry nodes determine their Globally Unique IDentifier (GUID) by applying a SHA-1 hash to a public key named in the Router argument list.

The TClient and DTClient stages are responsible for the building and maintenance of routing tables.  Where the DTClient provides functionality for a new Tapestry node to be added to an existing Tapestry network, the TClient represents the StaticTClient component corresponding to the bootstrap mechanism that allows multiple Tapestry nodes to initialize, join and form a Tapestry network.  Depending on the value of the dynamic_route init variable (either static or dynamic), a Tapestry node expects to either participate in a set of bootstrapping static Tapestry nodes, or else to join an existing Tapestry network.

In the static configuration, a node called the Federation is first started, followed by a small number of Tapestry nodes.  The Federation node runs a Federation stage which allows static Tapestry nodes to find each other and synchronize operations.  It reads from its configuration file the number of expected static Tapestry nodes.  As each static Tapestry node starts up, it sends a "hello" message to the Federation. When the Federation has heard from the expected number of static nodes, it sends each node a list of addresses for all static nodes.  Each static node then sends ping messages to all other static nodes, and uses the result to build its own routing table.  Once a static node has built its routing table, it sends a ready message to the Federation.  When the Federation has received ready messages from every static node, it sends each of them a begin message.  This acts as a global barrier, and tells static nodes when they can proceed.

Note that in an environment where a set of Tapestry nodes includes both static and dynamic nodes, only the static nodes need to be configured as above. The dynamic Tapestry nodes have dynamic_route set to dynamic, and use the dynamic insertion algorithm to insert themselves into a statically constructed Tapestry network.  Tapestry nodes in the original static network still have DTClient stages, which participate in the dynamic insertion of dynamic nodes.  Dynamic nodes need to specify a gateway argument that specifies the location of a node that the new node uses as an introduction gateway into the Tapestry (set the config argument gateway inside the DTClient stage to a IP address and port or DNS name and port).  Dynamic nodes should be instantiated only after the static Tapestry node has stabilized.

Tapestry Interface

As introduced in the background page , the basic Tapestry interface includes four simple message types.  In the current implementation, the Tapestry API is described by a set of abstract message classes under ostore.tapestry.api.  See the javadocs for more information on the API messages.

To write a Tapestry application, there are several key steps:
  1. Write the necessary messages to interface with Tapestry
  2. Write an eventHandler class that would serve as the application
  3. Write configuration file(s) to define stages and specify initialization arguments
To interface with Tapestry, an application needs to create it's own message types by extending (instantiating) the abstract API message types.  For instance, a chat client instance may use a message to publish the online status of a local user.  It could create something like this:
    public class ChatStatusMsg extends TapestryPublishmsg {

Any object or message that needs to go across the network layer to another Tapestry instance (whether the other Tapestry instance is on another physical machine or not) needs to implement the QuickSerializable interface. This interface includes two methods. The first is a constructor with an argument of type InputBuffer, and the second is the serialize(OutputBuffer) method. These two functions specify how the object is serialized into bytes, and also how it is reconstructed from a byte array. The OutputBuffer type recognizes most primitive types, and you can serialize fields by just calling buffer.add(object). More complex objects such as arrays or your own creations will have to be broken down into its primitive components and serialized that way. To deserialize the object, just call the deserialization functions such as InputBuffer.nextLong() and InputBuffer.nextDouble() to read in primitives, and reconstruct complex objects in the same order as they were deconstructed in serialize(OutputBuffer).  
public MyObjectType(InputBuffer buffer) throws QSException
public void serialize(OutputBuffer buffer)
Don't write a working type_code function. This function was part of a legacy interface. Instead, just write this:
public int type_code () {
    throw new NoSuchMethodError ();
}
Make sure your new message class is registered with the type table before it is received. The easiest way to do this is in an init function for your Sandstorm stage (see below), like this:
ostore.util.TypeTable.register_type ("my.class.Name");
Along with the core Tapestry Java files in ostore.tapestry.impl, I've included a simple Test program which the regression test uses to check for Tapestry's correctness.  The relevant source files are:
    Test.java
    TestFoundMsg.java
    TestReadyMsg.java
    TestClient.java
    TestLocateMsg.java
The test starts up 4 static Tapestry nodes into a Tapestry network.  The 4 nodes then publish a set of random objects. A new node is inserted dynamically.  The new nodes publishes some objects, tries to locate all of the published objects.  It then unpublishes its own objects, and confirms their deletion by another set of locate messages.  The relevant configuration files are:
    test-activeclient.cfg
    test-client.cfg
    test-federation.cfg

Note that these three config files contain variables in the place of actual values for items such as port numbers and key names.  The regression test in src/regression/test-ostore.tapestry.dynamic uses string replacement to generate the necessary number of configurations of each type, with the correct values substituted in.  When the test is run, it first generates a set of cfg files, then does a "run-sandstorm configfile " on each one to start a virtual node.   The useful perl functions are stored in the Toolbox.pm module in the regression directory.


Tapestry Configuration Parameters
Much of Tapestry's operational behavior is specified at start-up time by the use of arguments in Sandstrom configuration files. You can see examples of these configuration files by looking for .cfg files. Here, I'll explain a few of the configuration values.

Finally...

Ben Y. Zhao

July 2, 2003