[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: Does hadoop delete the intermediate data
From:       Jim Twensky <jim.twensky () gmail ! com>
Date:       2009-08-31 16:47:50
Message-ID: 7a8854060908310947n5784768byc6e00908882d33 () mail ! gmail ! com
[Download RAW message or body]

Hi Jeff,

The problem may also be related to the large log files if you use the
cluster for too many jobs. Check out your hadoop log directory and see
how big it is. You can decrease the maximum size of a log file using
one of the hadoop configuration files under conf.

Jim

On Mon, Aug 31, 2009 at 2:19 AM, Chandraprakash
Bhagtani<cpbhagtani@gmail.com> wrote:
> Hadoop does delete the intermediate data after the job completes.
> Jobtracker sends signal to Tasktracker to delete intermediate data
> when the job completes.
>
> The problem in your case might be some of your running job might not
> have been killed gracefully or Jobtracker failed for some reason.
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
>
> On Tue, Aug 25, 2009 at 6:19 AM, zhang jianfeng <zjffdu@gmail.com> wrote:
>
>> Hi all,
>>
>> I found my cluster’s space usage increase over time although I did not
>> upload new data.  And there's a lot of files under folder /tmp .
>>
>> So I guess hadoop won’t delete the intermediate data(output of mapper).
>>
>> Am I right ?
>>
>>
>> Thank you.
>>
>> Jeff zhang
>>
>

[prev in list] [next in list] [prev in thread] [next in thread]