[prev in list] [next in list] [prev in thread] [next in thread] 

List:       avro-dev
Subject:    Re: Avro serialization in streaming context
From:       Niels Basjes <Niels () basjes ! nl>
Date:       2018-02-23 21:24:41
Message-ID: CADoiZqrM0YTbdanG75FB4cx4Pyk6OzSFaW8Ent3TexiL4G=27g () mail ! gmail ! com
[Download RAW message or body]


HI,

In Avro 1.8.2 we have added a "Message" format specification.
In the Java classes there are now extra methods generated like
toByteBuffer() and fromByteBuffer() that are intended for exactly the
purpose of using Avro as the serialization specification in a streaming
context.
In fact at bol.com (where I work) we already do this in production.

Please have a look at this because I have the feeling this already does a
lot of what you mention.

Niels Basjes


On Thu, Feb 22, 2018 at 7:18 AM, Commeau, Gabriel <gabriel.commeau@gmail.com
> wrote:

> Hi all,
>
> I've been working for several years now on a streaming data platform, and
> we've been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I'm poking the community to
> see if there's an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There's also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven't seen a utility like that, but please correct me if I
> missed it. Then do you think it'll be a contribution that you'd welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes


[prev in list] [next in list] [prev in thread] [next in thread]