[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    RE: NullPointerException when running multiple reducers with Hadoop
From:       Aaron Baff <Aaron.Baff () telescope ! tv>
Date:       2011-06-30 17:02:59
Message-ID: DEC9971F0504A54784C9664B469E53F10307959F8840 () valkyrie ! telescope ! local
[Download RAW message or body]

Are you submitting from a Windows machine? I remember seeing at one point in time \
something like this dealing with trying to find the user & group on MR submission (if \
you look, it has to do with javax.security.auth and \
org.apache.hadoop.security.UserGroupInformation), and ran into this when submitting \
from a Windows machine.

--Aaron

-----Original Message-----
From: Paolo Castagna [mailto:castagna.lists@googlemail.com]
Sent: Wednesday, June 29, 2011 11:54 PM
To: common-user@hadoop.apache.org
Subject: NullPointerException when running multiple reducers with Hadoop \
0.22.0-SNAPSHOT

Hi,
I am using Apache Whirr to setup an Hadoop cluster on EC2 using Hadoop
0.22.0 SNAPSHOTs (nightly) builds from Jenkins. For details, see [1,2].
(Is there a better place where I can get nightly builds for Hadoop?)

I have a Reducer which does not emit any (key,value) pairs (it generates
only side effect files). When I run using only one reducer everything seems
fine. If I setup multiple reducers, each one generating different side effect
files, I get a NullPointerException:

  WARN  Exception running child : java.lang.NullPointerException
    at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:96)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:239)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:225)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
    at org.apache.hadoop.mapred.Child.main(Child.java:217)

Have you ever seen this exception/stacktrace?

My driver is here [3] and the reducer is here [4].

Regards,
Paolo

  [1] https://github.com/castagna/tdbloader3/blob/master/hadoop-ec2.properties
  [2]
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-22-Build/lastSuccessfulBuild/artifact/hadoop-0.22.0-SNAPSHOT.tar.gz
  [3]
https://github.com/castagna/tdbloader3/blob/master/src/main/java/com/talis/labs/tdb/tdbloader3/ThirdDriver.java
  [4]
https://github.com/castagna/tdbloader3/blob/master/src/main/java/com/talis/labs/tdb/tdbloader3/ThirdReducer.java



[prev in list] [next in list] [prev in thread] [next in thread]