[prev in list] [next in list] [prev in thread] [next in thread] 

List:       avro-user
Subject:    Re: Map having <string, Object>
From:       Scott Carey <scottcarey () apache ! org>
Date:       2011-12-07 18:01:42
Message-ID: CB04DF2A.62CBE%scott () richrelevance ! com
[Download RAW message or body]

The best practice is usually to use the flexible schema with the union
value rather than transmit schemas each time.  This restricts the
possibilities to the set defined, and the type selected in the branch is
available on the decoding side.  In the case above the number of variants
is not too large for this approach to be unwieldy, and there may be
benefits of knowing the type on the other side without inspecting the
value.

You can construct an Avro schema that represents all possible data
variants, effectively tagging the types of every field during
serialization using unions.  However none of the Avro APIs are (yet)
optimized for this use case, it would be somewhat clumsy to work with, and
is less space efficient.  Other serialization systems are a better fit for
completely open-ended data schemas.

One can look at Avro as a serialization system, but I see it more as a
system for describing your data.  It provides tools for describing and
transforming data that exists in related forms (e.g. older or newer schema
versions) to the form you want to see (e.g. current schema).  Whether this
data is serialized or an object graph is less important than that it
conforms to a schema.  A transformation between a serialized form and an
object graph is one use case of many possibilities.

Think about your use case from that perspective.  Ask whether this is data
that gains benefit from describing it with an Avro Schema and then
interpreting it as conforming to that schema.  If it is completely open
ended there may be little benefit and significant overhead.

You can also embed JSON or binary JSON in Avro data fairly easily using
Jackson JSON.


On 12/7/11 9:10 AM, "Gaurav Nanda" <gaurav324@gmail.com> wrote:

>I agree that in this case Json would be equally helpful. But In my
>application there is one more type of message, where untagged data can
>provide compact data encoding. So to maintain consistency, I preferred
>to send these kind of messages also using avro.
>
>@where untagged data can provide compact data encoding.
>In that case also, my schema has to be dynamically generated (i.e. on
>runtime), so has to be passed to client. So would avro be better to
>compressed json is that case?
>
>Thanks,
>Gaurav Nanda
>
>On Wed, Dec 7, 2011 at 9:17 PM, Tatu Saloranta <tsaloranta@gmail.com>
>wrote:
>> On Wed, Dec 7, 2011 at 5:16 AM, Gaurav <gaurav324@gmail.com> wrote:
>>> Hi,
>>>
>>> We have a requirement to send typed(key-value) pairs from server to
>>>clients
>>> (in various languages).
>>> Value can be one of primitive types or a map of same (string, Object)
>>>type.
>>>
>>> One option is to construct record schema on the fly and second option
>>>is to
>>> use unions to write schema in a general way.
>>>
>>> Problems with 1 is that we have to construct schema everytime
>>>depending upon
>>> keys and then attach the entire string schema to a relatively small
>>>record.
>>>
>>> But in second schema, u don't need to write schema on the wire as it is
>>> present with client also.
>>>
>>> I have written one such sample schema:
>>> 
>>>{"type":"map","values":["int","long","float","double","string","boolean"
>>>,{"type":"map","values":["int","long","float","double","string","boolean
>>>"]}]}
>>>
>>> Do you guys think writing something of this sort makes sense or is
>>>there any
>>> better approach to this?
>>
>> For this kind of loose data, perhaps JSON would serve you better,
>> unless you absolutely have to use Avro?
>>
>> -+ Tatu +-



[prev in list] [next in list] [prev in thread] [next in thread]