[prev in list] [next in list] [prev in thread] [next in thread]
List: avro-user
Subject: Re: Dynamic Package/namespace naming
From: Lewis John Mcgibbney <lewis.mcgibbney () gmail ! com>
Date: 2014-05-06 16:10:21
Message-ID: CAGaRif3Q9jzxRh0ueR8rPhNqcj+8buVmGpArOpfqN8TSAc8s4Q () mail ! gmail ! com
[Download RAW message or body]
Hi Sean thanks for the additional info. This is really helpful.
Some of these constraints are not what I would prefer... and of course mean
more work. Powers that be request them ;)
Thanks
On May 6, 2014 9:06 AM, "Sean Busbey" <busbey@cloudera.com> wrote:
> Hi Lewis,
>
> Well, I'm not sure what you're going to gain by isolating each file into
> its own namespace.
>
> However, one way you can achieve that is to take the information you want
> included from before and then create a hash of it. You can use the hex
> representation of the hash to give you something that
>
> * meets Avro's requirements for namespace names
> * uniquely identifies each namespace based on experiment, channel, and
> timestamp
>
> If you need the file to also contain the information that went into making
> that hash, you can attach it as Avro-ignored metadata on the schema.
>
> -Sean
>
>
> On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Sean,
>> Arggggh...
>> Maybe I can propose a change to the namespace semantics in my project.
>> I've been given the following brief.
>> Data files should have a logical namespace structure which can be
>> globally unified across multiple instruments and receiver platforms into a
>> common namespace on the filesystem.
>> I have a whole list of other data representation constraints and Avro
>> satisfies them all. This one is the only 'issue'.
>> Thanks for your help.
>> Lewis
>> On May 5, 2014 4:21 PM, "Sean Busbey" <busbey+lists@cloudera.com> wrote:
>>
>>> Hi Lewis!
>>>
>>> Avro namespaces don't allow the characters '/', ':', or '-'. So your
>>> specific example would not work. The allowed characters for a namespace
>>> are defined in the Avro spec[1].
>>>
>>> It would help if you could clarify what purpose namespacing serves in
>>> the system.
>>>
>>> [1]: http://avro.apache.org/docs/current/spec.html#Names
>>>
>>>
>>> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>>> Hi Charles
>>>> Thanks for reply.
>>>> Re: more info
>>>> Say I want my namespace to be
>>>>
>>>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>>>>
>>>> Where all constituent parts of namespace separated by '/' are
>>>> dynamic... is this possible via builder api?
>>>> Thanks again
>>>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
>>>> Charles.X.Pritchard.-ND@disney.com> wrote:
>>>>
>>>>> Need more info — you can use the Schema builder to do anything you
>>>>> like at runtime,
>>>>> Schema.createRecord and setFields.
>>>>>
>>>>>
>>>>>
>>>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>
>>>>> Hi Folks,
>>>>> I'm trying to propose Afro for a project I've been drafted on to.
>>>>> Namespace declarations are important... and a requirement is that
>>>>> namespaces are dynamic in nature... preferably even decided at runtime.
>>>>> Is this possible in Avro?
>>>>> Thanks
>>>>> Lewis
>>>>>
>>>>>
>>>>>
>>>
>
>
> --
> Sean
>
[Attachment #3 (text/html)]
<p dir="ltr">Hi Sean thanks for the additional info. This is really helpful.<br>
Some of these constraints are not what I would prefer... and of course mean more \
work. Powers that be request them ;)<br> Thanks</p>
<div class="gmail_quote">On May 6, 2014 9:06 AM, "Sean Busbey" <<a \
href="mailto:busbey@cloudera.com">busbey@cloudera.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr">Hi \
Lewis,<div><br></div><div>Well, I'm not sure what you're going to gain by \
isolating each file into its own namespace.</div><div><br></div><div>However, one way \
you can achieve that is to take the information you want included from before and \
then create a hash of it. You can use the hex representation of the hash to give you \
something that</div>
<div><br></div><div>* meets Avro's requirements for namespace names</div><div>* \
uniquely identifies each namespace based on experiment, channel, and \
timestamp</div><div><br></div><div>If you need the file to also contain the \
information that went into making that hash, you can attach it as Avro-ignored \
metadata on the schema.</div>
<div><br></div><div>-Sean<br><div class="gmail_extra"><br><br><div \
class="gmail_quote">On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <span \
dir="ltr"><<a href="mailto:lewis.mcgibbney@gmail.com" \
target="_blank">lewis.mcgibbney@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Hi Sean,<br> Arggggh...<br>
Maybe I can propose a change to the namespace semantics in my project.<br>
I've been given the following brief.<br>
Data files should have a logical namespace structure which can be globally unified \
across multiple instruments and receiver platforms into a common namespace on the \
filesystem.<br> I have a whole list of other data representation constraints and \
Avro satisfies them all. This one is the only 'issue'.<br> Thanks for your \
help.<span><font color="#888888"><br> Lewis</font></span></p><div><div>
<div class="gmail_quote">On May 5, 2014 4:21 PM, "Sean Busbey" <<a \
href="mailto:busbey%2Blists@cloudera.com" \
target="_blank">busbey+lists@cloudera.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Lewis!<div><br></div><div>Avro namespaces don't allow the \
characters '/', ':', or '-'. So your specific example would \
not work. The allowed characters for a namespace are defined in the Avro \
spec[1].</div>
<div><br></div><div>It would help if you could clarify what purpose namespacing \
serves in the system.</div><div><br></div><div>[1]: <a \
href="http://avro.apache.org/docs/current/spec.html#Names" \
target="_blank">http://avro.apache.org/docs/current/spec.html#Names</a></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 5, 2014 \
at 6:14 PM, Lewis John Mcgibbney <span dir="ltr"><<a \
href="mailto:lewis.mcgibbney@gmail.com" \
target="_blank">lewis.mcgibbney@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Hi Charles<br> Thanks for reply.<br>
Re: more info<br>
Say I want my namespace to be</p>
<p dir="ltr">experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/</p>
<p dir="ltr">Where all constituent parts of namespace separated by '/' are \
dynamic... is this possible via builder api?<br> Thanks again</p><div><div>
<div class="gmail_quote">On May 5, 2014 4:04 PM, "Pritchard, Charles X. \
-ND" <<a href="mailto:Charles.X.Pritchard.-ND@disney.com" \
target="_blank">Charles.X.Pritchard.-ND@disney.com</a>> wrote:<br \
type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <div style="word-wrap:break-word">Need more info — you can \
use the Schema builder to do anything you like at runtime,<div>Schema.createRecord \
and setFields.</div><div><br></div><div><div><div><br></div><div><br><div><div>
On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <<a \
href="mailto:lewis.mcgibbney@gmail.com" \
target="_blank">lewis.mcgibbney@gmail.com</a>> wrote:</div><br><blockquote \
type="cite"><p dir="ltr">Hi Folks,<br> I'm trying to propose Afro for a project \
I've been drafted on to.<br> Namespace declarations are important... and a \
requirement is that namespaces are dynamic in nature... preferably even decided at \
runtime.<br> Is this possible in Avro?<br>
Thanks<br>
Lewis</p>
</blockquote></div><br></div></div></div></div></blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div \
dir="ltr">Sean</div> </div></div></div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic