[prev in list] [next in list] [prev in thread] [next in thread]
List: avro-user
Subject: Re: Problem running an Non Avro to Avro Job
From: Chris Wilkes <cwilkes () gmail ! com>
Date: 2011-09-15 14:35:29
Message-ID: CAAJrG_kpQbA9h2P-Yu7=2CO__wY5W+nLWju+33S-Hn7RXCpJkw () mail ! gmail ! com
[Download RAW message or body]
In org.apache.avro.mapred.HadoopMapper:
public void collect(OUT datum) throws IOException {
if (isMapOnly) {
// don't require a Pair for datum
} else {
// cast datum to a pair
}
}
so in your case you probably just want to set the number of reducers
to 0 to create an avro record without a Pair as the output value (you
mention "key" below btw).
Actually re-reading it looks like you are trying to make a Pair
object. Why are you putting that inside an AvroWrapper? Just do
collector.collect(p)
And set your map output schema to TestPair.SCHEMA$ where SCHEMA$ =
Pair.getPairSchema(keySChema, valueSchema)
Chris
On Thu, Sep 15, 2011 at 6:10 AM, Rohini U <rohini.u@gmail.com> wrote:
> Hi All,
>
> I am using avro 15.3 and CDH3. I have a query in running a Non Avro to Avro
> Job. My mapper is non Avro mapper and it outputs a avro data.
> When I run this, I get an error saying not a Pair Schema. Does the key
> always have to be a Pair Schema?
>
> public void map(Text key, Text value,
> OutputCollector<AvroWrapper<TestPair>, NullWritable> output,
> Reporter reporter) throws IOException {
> String valueString = value.toString();
> String[] toks = valueString.split("\t");
> TestPair p = new TestPair();
> p.left = toks[0];
> p.right = toks[1];
>
>
> output.collect(new AvroWrapper<TestPair>(p),wt);
> }
>
> Thanks,
> -Rohini
>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic