[prev in list] [next in list] [prev in thread] [next in thread] 

List:       flume-dev
Subject:    Re: Breaking up Flume (again)
From:       Ralph Goers <ralph.goers () dslextreme ! com>
Date:       2023-02-26 18:50:34
Message-ID: C7F000B4-B981-406F-9E9F-8445C8146B33 () dslextreme ! com
[Download RAW message or body]

The morphline solr sink has a dependency on Kite, which is a project abandoned by \
Cloudera. Someone would have to copy the relevant parts into the morphline repo and \
maintain them there. I have no interest myself in doing that.

I already split the Elasticsearch sink into the flume-search repo. As I recall I had \
problems building it. We have discussed that in other emails. It needs to be \
upgraded. I suspect the API we would have to use has an acceptable license but I \
believe ES itself has licensing problems. 

To be honest, I don't know what the deal is with the legacy sources and why we even \
have them. We have an Avro source and Thrift source in Flume Core so I don't know why \
we even keep them around.

I personally don't use Hadoop or any of its related technology. While I know those \
are important, it is likely I personally will only apply PRs to any of them.

Ralph

> On Feb 26, 2023, at 10:29 AM, Bessenyei Balázs Donát <bessbd@apache.org> wrote:
> 
> +1.
> 
> For #3, which ones do you think can no longer be practically supported?
> 
> 
> Donat
> 
> On Sun, Feb 26, 2023 at 8:08 AM Ralph Goers <ralph.goers@dslextreme.com> wrote:
> > 
> > As I mentioned last year I would like to start breaking up flume into separate \
> > repos. There are a few reasons for this: 1. Flume has grown so large that the CI \
> > system can no longer build it. The jobs run out of disk space due to the large \
> > logs. 2. The build takes a very long time to run.
> > 3. There are several components that can no longer be practically be supported.
> > 
> > To this end I am planning on creating the following Git repos:
> > flume-hadoop
> > flume-http
> > flume-irc
> > flume-jdbc
> > flume-jms
> > flume-kafka
> > flume-kudu
> > flume-legacy
> > flume-morphline
> > flume-scribe
> > flume-search
> > flume-spring-boot
> > flume-twitter
> > 
> > For the time being I would propose everything else remain in the current Flume \
> > repo. 
> > Note that as each of these is populated they will each need to be released, \
> > However, most of these are fairly inactive so after the initial release they may \
> > not need to be touched very often. 
> > Also, since Jira now requires new users to request us to create accounts for them \
> > I would propose that as each of these repos are set up that they be configured to \
> > enable GitHub Issues. 
> > I am looking for feedback on this but if I don't get any I plan to start work on \
> > this within a week or so. 
> > Ralph


[prev in list] [next in list] [prev in thread] [next in thread]