GM static mapping

The default way to keep the route tables in Myrinet cards up to date is to run an "active" mapper continuously on a single node. This is a bad idea for a number of reasons:

It doesn't work. Code bugs, perhaps.
Another single point of failure. There is provision for a "passive" mapper on another node to take over for a failed active mapper, if it would work.
Route flap. In-flight messages are dropped during a reconfiguration to avoid deadlock.
Scalability. Distributed mapping does not scale.

There are three pieces of activity that must occur to configure a GM Myrinet network:

Determine topology.
Generate routes.
Install routes in each node.

If you know the topology of your network (which cables are plugged into which ports), you can type it in directly. Or you may find it expedient to use GM's mapper code to automatically discover the topology.

Next, given the topology, it is necessary to generate deadlock-free routes for each pair of nodes. The classic algorithm is up*/down*, and is implemented in the GM simple_routes code. In the case of regular networks it may be possible to generate shorter routes by using a specific algorithm for that network. See mt/ad_hoc_calculator for examples.

The first two steps above have to happen only once per reconfiguration. If cables get moved around, or hostnames change, or nodes get added to the cluster, it is necessary to regenerate the routes.

As each node in the cluster boots, it can install the routes file generated in the first two steps. This can happen independently of any other machine in the cluster.

Caveats

If host names change or cables are moved, the routes must be rebuilt. If you have managed to build a network with multiple paths between switches, and one of those interlink switches fails, fail-over to another link will not happen automatically as it might in an active mapping setup.

GM put randomization in the route generation algorithm to avoid placing the root node in the same location every time. The up*/down* algorithm arranges all the nodes in a tree, connected by the links, and picks a direction for each link. Routes are generated which first go "up" the tree, then "down". This leads to more traffic on links near the top of the tree. This added traffic may be significant depending on the topology and the message distribution on the network, but we haven't noticed any degraded performance by not having the benefit of this randomization.

OSC

Here are the scripts we use at OSC to implement this.

static.args Input file to standard mapper program, specifies -map-once and names of output files.
run-static-mapper Little script to invoke the mapper. Just do "mapper static.args" by hand on one of the machines with Myrinet if you prefer.
load-routes This is invoked at boot for every node from /etc/rc.d/rc.local. It invokes the file_mapper to install the routes from a file. We have two Myrinet cards in each node, hence the loop. One-card installations can just drop the -unit argument.

Building

Note that the GM distribution does not build the file_mapper code by default. In the mt/ directory of the GM source, type make all gm, then install by hand the executable tools/$arch/file_mapper.

Or, if you are so driven, you can generate only the file_mapper executable with the following lines typed to a shell from the GM source directory:

( cd mt/file_calculator ; make )
( cd mt/smapper ; make )
( cd mt/deadlock ; make )
( cd mt/libmt ; mkdir -p intel_linux ; make intel_linux/libmt{,gm}.a )
( cd mt/tools ; mkdir -p intel_linux ; make intel_linux/file_mapper )
cp mt/tools/intel_linux/file_mapper binary/sbin

This expects that the rest of GM has been build as usual, and assumes one particular architecture. Then you still must go and install the contents of binary/ somehow. (We do this by hand with something like cp -R binary/* /usr/local/gm-1.4pre51.)

Back to index

Last updated 18 Mar 2005.