[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: Introducing BDS: A datacenter scripting language
From: Marco Massenzio <marco () mesosphere ! io>
Date: 2015-05-15 23:04:54
Message-ID: CAD2Zrn3QbS+s7dw8G_xqEEUp7s99mtgbXcXUdRPD=ouSG2CXGQ () mail ! gmail ! com
[Download RAW message or body]
That's awesome, Pablo - will definitely be fooling around with it!
Thanks for using Mesos, BTW - always good to see folks building cool stuff
on top of it :)
*Marco Massenzio*
*Distributed Systems Engineer*
On Thu, May 14, 2015 at 6:45 PM, Pablo Cingolani <
pablo.e.cingolani@gmail.com> wrote:
>
> Hi Everyone,
> I've been working on a simple programming language to create large
> data pipelines on Mesos. The language is called BDS which stands
> for BigDataScript (yes, the name is kind of a joke for all jargon-lovers
> out there) and here is the web page:
>
> http://pcingola.github.io/BigDataScript/
>
> Needles to say, it's open source and the code is available is GitHub.
> At the moment I'm using BDS mostly for analysis of large genetic datasets
> on our 25,000 core cluster, but it should scale to large(er) clusters as
> well.
>
> BDS has a few interesting features:
> - Runs on Mesos (obviously) as well as SunGridEngine, Torque,
> MOAB, a large server or just your laptop.
>
> - You can develop on your laptop (without having to install Mesos or
> any cluster management system) and then deploy your script to a
> Mesos
> cluster/datacenter without modification.
>
> - It performs automatic task dependency and schedules tasks according
> to
> the implicit (or explicit) DAG.
>
> - It has lazy processing. Checks whether performing a task is
> necessary and
> skips tasks whose output does not need to be updated (make-style).
>
> - It performs automatic checkpointing and has absolute serialization,
> so you
> can copy the checkpoint file to another computer and continue
> running
> exactly where you left.
>
> - It can handle several parallel pipeline branches (threads).
>
> - Allows to define DAGs in a declarative form (using 'goals').
>
> - Cleans up stale files (and queues tasks in non-Mesos cluster).
>
> Other cool features:
>
> - Automatically parses command line options in your scripts (it also
> creates "help" for you)
> - Logs all processes's stdout / stderr and exit status
> - It has a built in debugger
> - It has a built in unity testing framework
>
> You can read more about all these features here:
>
> http://pcingola.github.io/BigDataScript/bigDataScript_manual.html
>
> I hope you find it useful and please do send me any
> feedback you have.
> Yours
>
> Pablo
>
>
>
>
[Attachment #3 (text/html)]
<div dir="ltr">That's awesome, Pablo - will definitely be fooling around with \
it!<div>Thanks for using Mesos, BTW - always good to see folks building cool stuff on \
top of it :)</div></div><div class="gmail_extra"><br clear="all"><div><div \
class="gmail_signature"><div dir="ltr"><i><font color="#0b5394" \
style="background-color:rgb(255,255,255)">Marco Massenzio</font></i><div><i><font \
color="#6fa8dc">Distributed Systems Engineer</font></i></div></div></div></div> \
<br><div class="gmail_quote">On Thu, May 14, 2015 at 6:45 PM, Pablo Cingolani <span \
dir="ltr"><<a href="mailto:pablo.e.cingolani@gmail.com" \
target="_blank">pablo.e.cingolani@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><br><div>Hi Everyone,</div><div> I've \
been working on a simple programming language to create large </div><div>data \
pipelines on Mesos. The language is called BDS which stands </div><div>for \
BigDataScript (yes, the name is kind of a joke for all jargon-lovers </div><div>out \
there) and here is the web page:</div><div><br></div><div> <a \
href="http://pcingola.github.io/BigDataScript/" \
target="_blank">http://pcingola.github.io/BigDataScript/</a></div><div><br></div><div> \
Needles to say, it's open source and the code is available is \
GitHub.</div><div>At the moment I'm using BDS mostly for analysis of large \
genetic datasets </div><div>on our 25,000 core cluster, but it should scale to \
large(er) clusters as well.</div><div><br></div><div> BDS has a few interesting \
features:<br></div><div> - Runs on Mesos (obviously) as well as SunGridEngine, \
Torque, </div><div> MOAB, a large server or just your \
laptop.</div><div><br></div><div> - You can develop on your laptop (without \
having to install Mesos or </div><div> any cluster management system) and \
then deploy your script to a Mesos </div><div> cluster/datacenter without \
modification.</div><div><br></div><div> - It performs automatic task dependency \
and schedules tasks according to </div><div> the implicit (or explicit) \
DAG.</div><div><br></div><div> - It has lazy processing. Checks whether \
performing a task is necessary and </div><div> skips tasks whose output does \
not need to be updated (make-style).</div><div><br></div><div> - It performs \
automatic checkpointing and has absolute serialization, so you </div><div> \
can copy the checkpoint file to another computer and continue running </div><div> \
exactly where you left.</div><div><br></div><div> - It can handle several \
parallel pipeline branches (threads).</div><div><br></div><div> - Allows to \
define DAGs in a declarative form (using 'goals').</div><div><br></div><div> \
- Cleans up stale files (and queues tasks in non-Mesos \
cluster).</div><div><br></div><div>Other cool features:</div><div><br></div><div> \
- Automatically parses command line options in your scripts (it also creates \
"help" for you)</div><div> - Logs all processes's stdout / \
stderr and exit status</div><div> - It has a built in debugger</div><div> \
- It has a built in unity testing framework</div><div><br></div><div> You can read \
more about all these features here: </div><div><br></div><div> <a \
href="http://pcingola.github.io/BigDataScript/bigDataScript_manual.html" \
target="_blank">http://pcingola.github.io/BigDataScript/bigDataScript_manual.html</a></div><div><br></div><div> \
I hope you find it useful and please do send me any </div><div>feedback you \
have.</div><div> Yours</div><span class="HOEnZb"><font \
color="#888888"><div><br></div><div> \
Pablo</div><div><br></div><div><br></div><div> </div></font></span></div> \
</blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic