[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: custom writablecomparable with complex fields
From:       Harsh J <harsh () cloudera ! com>
Date:       2013-08-31 20:52:59
Message-ID: CAOcnVr3LzwfddmAsCzVxk+d9y0Y=om50OVdVYeXSmUj-95DiJQ () mail ! gmail ! com
[Download RAW message or body]

The idea behind write(…) and readFields(…) is simply that of ordering.
You need to write your custom objects (i.e. a representation of them)
in order, and read them back in the same order.

An example way of serializing a list would be to first serialize the
length (so you know how many you'll be needed to read back), and then
serialize each item appropriately, using delimiters, or using
length-prefixes just like lists.

Mainly, you're required to tackle the serialization/deserialization on your own.

This is one of the reasons I highly recommend using a library like
Apache Avro instead. Its more powerful, faster, and yet simple to use:
http://avro.apache.org/docs/current/gettingstartedjava.html and
http://avro.apache.org/docs/current/mr.html. It is also popular and
carries first-grade support on several other hadoop-ecosystem
projects, such as Flume and Crunch.

On Sun, Sep 1, 2013 at 1:23 AM, Adeel Qureshi <adeelmahmood@gmail.com> wrote:
> I want to write a custom writablecomparable object with two List objects
> within it ..
>
> public class CompositeKey implements WritableComparable {
>
> private List<JsonKey> groupBy;
> private List<JsonKey> sortBy;
> ...
> }
>
> what I am not sure about is how to write
>
> readFields and write methods for this object. Any help would be appreciated.
>
> Thanks
> Adeel



-- 
Harsh J
[prev in list] [next in list] [prev in thread] [next in thread]