As mentioned on viff-devel, I've uploaded my benchmarking code on
bitbucket.org as a stand-alone application using VIFF as a library:

http://bitbucket.org/tpj/viffbench

My code only contains a few sample benchmarks. We still need to write
some good benchmarks .... but it should be easier now.

I have some benchmarking code ready that I will set up when we get
access to the new test servers. They should be available within a few
days.

I don't think my code is ready to be included in VIFF, but I plan to
set it up on the servers anyway when I get the time.

We should somehow coordinate the scripts running on the servers: One
of the reasons for using them is to ensure that only one thing is
running at the servers at a time.

Guys with scripts (Mikkel and Sigurd, I think): could you please add
your scripts to a /contrib directory in VIFF?

Just add some comments on how they should be used and put them there.
I guess a line in the README file would be okay to point people to them.

Marked as in-progress, as Martin and I work on this issue.

I think it's interesting even though it's only for 3 players. Combined
with Mikkels script

  http://tracker.viff.dk/file6/testmillionaires.bat

we could have scripts for both main platforms.

My scripts is only running 3 processes, and only locally, so it might 
not be interesting for benchmarking purposes, but it is really 
convenient for the edit/debug/run-cycle.

It runs each process in its own xterm, and they can all conveniently 
be shut down by Ctrl-c'ing the mother process.

run like:

./run.bash sort.py -n 15

Adding Sigurd to the nosy list -- I think you have a script as well?

The launcher needs a file with hostnames. I'm attaching the file I've
been using at DAIMI.

My latest attempt at an automatic launcher. It requires that all
machines have the same filesystem layout.

Renamed since this is more about the automatic benchmarks. Graphs
would be extra bells and whistles.

We really want this...

To get the most correct benchmark results we should run the tests on
different machines. The Fabric tool might be good for that:

  http://www.nongnu.org/fab/

It is meant for deploying (web)applications on (web)servers.

With the latest changes it is possible to reliably start and stop
programs which use VIFF -- that is a first step towards making this
kind of automation a reality.

To add some more info: the Matplotlib library would probably be a nice
tool for generating the graphs:

  http://matplotlib.sourceforge.net/

I have used it before -- with great success -- to automatically
generate nice-looking graphs.

I think we should build a system that will benchmark the code and
publish nice graphs with the results.

The tricky part is to do this in a reliable and meaningful way. The
easiest (most reliable) way would be to run three players on a single
machine. But that will probably not produce meaningful results since
the network delays are eliminated and the players will fight with each
other for the CPU (unless there are a core for each).

But maybe we should start with such a simple setup anyway -- I hope
that any performance loses or gains there will translate (somehow) to
loses or gains in a more realistic setting.