[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: Passing files and directory structures to the map reduce cluster via hadoop streaming?
From:       Paul Ingles <paul () oobaloo ! co ! uk>
Date:       2011-06-29 6:59:35
Message-ID: 4115A25C-9B57-4C7D-A6DE-4030B50FB324 () oobaloo ! co ! uk
[Download RAW message or body]

Hi,

I'm not familiar with wukong, but Mandy has some scripts that wrap the hadoop \
commands- the default behaviour IIRC is to package the folder the script is in.

This is then distributed so the app carries all its dependencies with it.

Happy to hear -files works for you.

Sent from my iPhone

On 29 Jun 2011, at 07:44, Guang-Nan Cheng <chenggn@gmail.com> wrote:

> Well, my bad. I made a simple test and confirmed that  -files works that way
> already.
> 
> For the two guys that "answered" my question, sorry I asked the question
> unclearly... I don't see how those two projects related to the question,
> but thank you. :D
> 
> 
> 
> 
> On Wed, Jun 29, 2011 at 12:35 AM, Abhinay Mehta <abhinay.mehta@gmail.com>wrote:
> 
> > We use Mandy: https://github.com/forward/mandy for this.
> > 
> > 
> > On 28 June 2011 17:26, Nick Jones <nick.jones@amd.com> wrote:
> > 
> > > Take a look at Wukong from the guys at Infochimps:
> > > https://github.com/mrflip/**wukong <https://github.com/mrflip/wukong>
> > > 
> > > 
> > > On 06/28/2011 11:19 AM, Guang-Nan Cheng wrote:
> > > 
> > > > I'm fancied about passing a whole ruby app to streaming, so I don't need
> > > > to
> > > > bother with ruby file dependencies.
> > > > 
> > > > For example,
> > > > 
> > > > ./streaming
> > > > 
> > > > ...
> > > > -mapper 'ruby aaa/bbb/ccc'
> > > > -files  aaa<--- pass the folder
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Is this supported already? If not, any tips on how to make this work?
> > I'm
> > > > willing to add some code by myself and rebuild the streaming jar.
> > > > 
> > > 
> > > --
> > > Nick Jones
> > > 
> > > 
> > > 
> > 


[prev in list] [next in list] [prev in thread] [next in thread]