[prev in list] [next in list] [prev in thread] [next in thread] 

List:       beowulf
Subject:    MoBiDiCK update...
From:       Christopher Hogue hogue () mshri ! on ! ca
Date:       1999-10-24 19:13:19
[Download RAW message or body]

rbm wrote:
> 
> hi
> 
> Does someone use Mobidick ? I believe it's works on the same matter as
> Mosix ? or not
> who nows of it is fast ?
> 
> Roy


Hi Roy,

MoBiDiCK is a project still being developed in my laboratory. It
uses http protocol based messaging, and programs reside on nodes behing
a web-server as CGIs (or FastCGIs). So it is not really like MOSIX, more
like PVM but
with Apache or any other web server as the daemon.  It also has message
passing formalisms that are based on http protocols for passing
parameters to CGI modules. Programming the clients has an MPI flavour,
(you link in a library that has the message passing API in it, data
collection, status and state keepers, garbage file cleanup, etc)
Different platforms are supported of course including Linux but also 
other Unixes and (gasp) NT.  

The front-end is all web-forms based, and we are trying to design it so
that biologists (non-cluster gurus!) can run stuff easily from a web
browser.

Anyway, we think it is a cool idea, and are pushing ahead on it,
methodically, with lots of testing because we are very concerned about
controlling the beast (we have a whale for a logo).  For example, it is
essential that we can kill accidental launches when a user puts in
incorrect parameters (runaway whale!).  This requires some careful
coding for the clients too, the clients have to be very robust and have
good error handling.  Every file written out by the client has to be
tracked for cleanup by the API, as well.  It is still an experimental
system and almost out of "alpha".  We have a couple of test compute
clients that work ith it, mostly molecular structure applications.  It
is best limited to problem domains that are compute bound, rather than
I/O bound, especially when clustering beyond your local Beowulf out into
the Internet.  We are using it to generate Monte Carlo protein
structures for a protein folding system we are building.

We have gotten well past proof of principle, both in running clusters
locally and over the Internet.  We have run it on a split cluster with
10 nodes in Toronto and 6 nodes in Calgary, so we know it has the
capability to do distribted clustering. We are trying to set it up so
that it ends up very user friendly and easy to configure.  It is still
not that polished and requires a demo run by us to see it in operation.

The main module of MoBiDiCK, a "dispatcher", uses a real-time database
of
processors, and there are load-balancing and parameter splitting
algorithms in it, to distribute tasks to available processors.  
Processor nodes run compute jobs as CGIs (or FastCGIs), and these call
back to a "collector" module that picks up data, so that everything ends
up on the starting computer, with all the required node data collection
and garbage file cleanup.

Recently we figured out how to conceputally chain together the main
dispatcher modules so that we can use other people's Beowulfs that are
on non-public IP addresses, as most are.  In this way the gateway node
(the one with public and non-public IP addresses) is set up as a
dispatcher "child", that is called by a parent MoBiDiCK dispatcher. 
Fortunately we already designed the dispatcher as a CGI itself (i.e. it
is callable...), so it just needs a new parameter wrapper, so that one
dispatcher can call another.  A whole Beowulf cluster would then appear
as a multiprocessor machine with one IP address to the master MoBiDiCK
system.  The local cluster owner just has to set up a local processor
database with the private IPs/URLs of their nodes.  This part requires a
bit more work on the system, but it will get us to clusters of clusters
over the Internet.

We have just now submitted our first abstract about it in which we show
proof of principle, node scaling and efficiency.  We'll try to put some
more up to date information about timing and efficiency on our site,
I'll send a message to this group when that is done, but for now, stay
tuned!

Cheers.

Christopher Hogue

---------------------------------------
Christopher W.V. Hogue, Ph.D.
Samuel Lunenfeld Research Institute
Mt. Sinai Hospital 600 University Ave.
Toronto Ontario Canada M5G 1X5
(416) 586-4800 xt2866
fax (416) 586-8857

hogue@mshri.on.ca

http://bioinfo.mshri.on.ca
-------------------------------------------------------------------
To unsubscribe send a message body containing "unsubscribe"
to beowulf-request@beowulf.org

[prev in list] [next in list] [prev in thread] [next in thread]