[prev in list] [next in list] [prev in thread] [next in thread] 

List:       avro-dev
Subject:    Re: Avro Tranformation Language / Avro Graph
From:       Anirudha Jadhav <anirudha () nyu ! edu>
Date:       2015-08-31 20:27:18
Message-ID: CAG7XKRhv1Bsc0B22HZFL53mBXWkUnE76eyHWE3NoaF51uw4Ueg () mail ! gmail ! com
[Download RAW message or body]


I need to go through approval process to release source, I wanted some feed
back while that process in in progress. Let me know if you have any
question or comments that I can address

- Anirudha Jadhav


On Mon, Aug 31, 2015 at 4:09 PM, Prajwal Tuladhar <praj@infynyxx.com> wrote:

> Hi Anirudha,
>
> Is it possible to view the source of those projects?
>
> On Mon, Aug 31, 2015 at 7:46 PM, Anirudha Jadhav <anirudha@nyu.edu> wrote:
>
>> Hello All,
>>
>> I would like to introduce you all to a project which we have been working
>> on using Avro and get some feedback.
>>
>> 1. AvroGraph
>> ------------
>> We have created an avro to graphml serializer / deserializer. This allows
>> us to visualize avro schemas in a graph to understand the relation between
>> all the data points. This will later lead to creation of lineage graphs
>> among other things
>> - Implementation
>>   o similar to json serializer / deserializer
>>   o Apache Tinkerpop is used as a graph library and can be used to
>> persist to a variety of graph stores.
>>   o support for scheme evolution between multiple version of the avro
>> schemas
>>   o lot of unit tests and documentation
>>
>> 2. Avro Transformation Language
>> -------------------------------
>> This is YAML based specification that will transform a data in a source
>> schema to a target schema. For this we introduce a "transform node" to join
>> the two schemas
>>  - The following operations can be done during the source to target data
>> transformations
>>    o Copy source leaves to target leaves
>>    o Copy source parent nodes to target parent nodes, only if the sub
>> graphs have the same structure.
>>    o Concatenate source nodes and copy to a target node
>>    o User-defined operations on the transforms
>>    o Extract certain leaves from the source and call an external end
>> point for data manipulation  eg. Spark / Http
>>
>>
>> Let me know how/if these components would benefit the apache avro project
>> and accordingly we would like to contribute it to the apache avro project.
>>
>> -Ani
>>
>
>
>
> --
> --
> Cheers,
> Praj
>



-- 
Anirudha P. Jadhav


[prev in list] [next in list] [prev in thread] [next in thread]