[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    No Reducer scenarios
From:       ☼ R Nair (रविशंकर नाठ<ra
Date:       2017-01-30 5:31:48
Message-ID: CAB1GMvmfy6PQ-XgjB7mDDsw-S8nzxnD9M_4v64EpQbu29jksJg () mail ! gmail ! com
[Download RAW message or body]

Dear all,


1) When we don't set the reducer class in driver program, IdentityReducer
is invoked.

2) When we set setNumReduceTasks(0), no reducer, even IdentityReducer is
invoked.

Now, in the second scenario, we observed that the output is part-m-xx
format(instead of part-r-xx format) , which shows the map output. But we
know that the output of Map is always written to intermediate local file
system. So who/which class is responsible for taking these intermediate Map
outputs from local file system and writes to HDFS ? Does this particular
class performs this write operation only when setNumReduceTasks is set to
zero?

Best, Ravion

[Attachment #3 (text/html)]

<div dir="ltr">Dear all,<div><br></div><div><br></div><div>1) When we don&#39;t set \
the reducer class in driver program, IdentityReducer is \
invoked.</div><div><br></div><div>2) When we set setNumReduceTasks(0), no reducer, \
even IdentityReducer is invoked.</div><div><br></div><div>Now, in the second \
scenario, we observed that the output is part-m-xx format(instead of part-r-xx \
format) , which shows the map output. But we know that the output of Map is always \
written to intermediate local file system. So who/which class is responsible for \
taking these intermediate Map outputs from local file system and writes to HDFS ? \
Does this particular class performs this write operation only when setNumReduceTasks \
is set to zero?</div><div><br></div><div>Best, Ravion</div></div>



[prev in list] [next in list] [prev in thread] [next in thread]