[prev in list] [next in list] [prev in thread] [next in thread] 

List:       avro-user
Subject:    Concatenate multiple files?
From:       Chris Miller <cmiller11101 () gmail ! com>
Date:       2015-11-26 2:41:20
Message-ID: CAN-NQFZN-ii19R+Betuz3_MwRQPzK95XUrxq-tfUyuDN94hQOA () mail ! gmail ! com
[Download RAW message or body]

I'm storing data generated from my web application in Apache Avro format.
The data is serialized and sent to an Apache Kinesis Firehose that buffers
and writes the data to Amazon S3 every 300 seconds or so. Since I have
multiple web servers, this results in multiple blobs of Avro files being
sent to Kinesis, upon which it concatenates and periodically writes them to
S3.

When I grab the file from S3, I can't using the normal Avro tools to decode
it since it's actually multiple files in one. I could add a delimiter I
suppose, but that seems risky in the event that the data being logged also
has the same delimiter.

What's the best way to deal with this? I couldn't find anything in the
standard that supports multiple Avro files concatenated into the same file.

--
Chris Miller

[Attachment #3 (text/html)]

<div dir="ltr"><div>I&#39;m storing data generated from my web application in Apache \
Avro format. The data is serialized and sent to an Apache Kinesis Firehose that \
buffers and writes the data to Amazon S3 every 300 seconds or so. Since I have \
multiple web servers, this results in multiple blobs of Avro files being sent to \
Kinesis, upon which it concatenates and periodically writes them to \
S3.</div><div><br></div><div>When I grab the file from S3, I can&#39;t using the \
normal Avro tools to decode it since it&#39;s actually multiple files in one. I could \
add a delimiter I suppose, but that seems risky in the event that the data being \
logged also has the same delimiter.</div><div><br></div><div>What&#39;s the best way \
to deal with this? I couldn&#39;t find anything in the standard that supports \
multiple Avro files concatenated into the same file.</div><div><br></div><div><div \
class="gmail_signature"><div dir="ltr"><div>--</div><div>Chris \
Miller</div></div></div></div> </div>



[prev in list] [next in list] [prev in thread] [next in thread]