[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Using custom Input Format
From:       Rakhi Khatwani <rkhatwani () gmail ! com>
Date:       2010-01-29 13:16:36
Message-ID: 2aa3aff81001290504k5d790953kd9c818248a36e4a8 () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


Hi,
        i have been trying to implement custom input and output formats. i
was successful enough in creating an custom output format. but whn i call a
mapreduce method which takes in a file, using custom input format, i get an
exception.
java.lang.NullPointerException
 at Beans.Content.write(Content.java:54)
 at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
  at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
  at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
 at CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:32)
 at CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:1)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
 at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
0 Hello World0 Hello World0 Hello World0

id:: null media:: null url:: null content::null

id:: null media:: null url:: null content::null
java.io.IOException: Job failed!
 at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
 at
CustomInputFormat.SampleCustomInputMapReduce.run(SampleCustomInputMapReduce.java:53)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at
CustomInputFormat.SampleCustomInputMapReduce.main(SampleCustomInputMapReduce.java:59)


I have attached the following files
Content => Custom object that implements writercomparible
ContentInputFormatV2 => inputformat which implements SequenceFileInputFormat
ContentRecordReader => implementation of RecordReader (Not required though,
it should work w/o it i assume).
SampleCustomInputMap => mapper class
SampleCustomInputReduce => reducer class
SampleCustomInputMapReduce => class which contains the main method and the
job configurations.
data and index => my inputfiles for the main function
For the 1st record it works fine, but for next record and records after
that, i get null. where could i go wrong??
Regards,
Raakhi


[Attachment #5 (text/html)]

<div>Hi, <br></div><div>        i have been trying to implement custom input and \
output formats. i was successful enough in creating an custom output format. but whn \
i call a mapreduce method which takes in a file, using custom input format, i get an \
exception. </div> <div></div><div>java.lang.NullPointerException<br> at \
Beans.Content.write(Content.java:54)<br> at \
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)<br> \
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)<br>
  at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)<br> \
at CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:32)<br> at \
CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:1)<br>  at \
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)<br> at \
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)<br> at \
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)<br>0      \
Hello World0      Hello World0     Hello World0<br> <br>id:: null media:: null url:: \
null content::null<br><br>id:: null media:: null url:: null \
content::null<br>java.io.IOException: Job failed!<br> at \
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)<br> at \
CustomInputFormat.SampleCustomInputMapReduce.run(SampleCustomInputMapReduce.java:53)<br>
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)<br> at \
CustomInputFormat.SampleCustomInputMapReduce.main(SampleCustomInputMapReduce.java:59)<br><br><br></div><div></div><div>I \
have attached the following files</div> <div></div><div>Content =&gt; Custom object \
that implements writercomparible</div><div>ContentInputFormatV2 =&gt; inputformat \
which implements SequenceFileInputFormat</div><div>ContentRecordReader =&gt; \
implementation of RecordReader (Not required though, it should work w/o it i \
assume).</div> <div>SampleCustomInputMap =&gt; mapper \
class</div><div>SampleCustomInputReduce =&gt; reducer \
class</div><div>SampleCustomInputMapReduce =&gt; class which contains the main method \
and the job configurations.</div><div>data and index =&gt; my inputfiles for the main \
function</div> <div>For the 1st record it works fine, but for next record and records \
after that, i get null. where could i go \
wrong??</div><div></div><div>Regards,</div><div>Raakhi</div><div></div>

--0016e64cd24c099f3a047e4d47a0--



[prev in list] [next in list] [prev in thread] [next in thread]