[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: conf.setNumReduceTasks(1) but the code called 3 times
From:       Mark Kerzner <markkerzner () gmail ! com>
Date:       2009-07-31 0:48:32
Message-ID: c9b0d8bd0907301748k3ff0adddt400ca0702d44f5f () mail ! gmail ! com
[Download RAW message or body]


Agree! And on Amazon they also told me that it is because I have a Combiner
that I am getting reduce() called many times. I switched the combiner off
and everything worked as expected.
Amazon guys watch this list too!

Mark

On Thu, Jul 30, 2009 at 11:33 AM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Another rule of thumb is to avoid side effects!
>
> On Thu, Jul 30, 2009 at 7:54 AM, Jason Venner <jason.hadoop@gmail.com
> >wrote:
>
> > A rule of thumb is to not enable speculative execution if the tasks have
> > side effects that are not cleaned up on task abort.
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>


[prev in list] [next in list] [prev in thread] [next in thread]