[prev in list] [next in list] [prev in thread] [next in thread]
List: log4j-dev
Subject: Re: Idea of Cloud as target for the bulk of log file
From: Tushar Kapila <tgkprog () gmail ! com>
Date: 2013-03-29 16:48:33
Message-ID: 5155C5C0.3030000 () gmail ! com
[Download RAW message or body]
From what I could see you plan to edit the rolling file appender so it
can take a custom action on some events (roll over when already rolled
over files is > X) at which point files are pushed to another system.
Rolling file appender would continue to do what it does natively. I
think a better design would be to enhance the appender in a generic way
so it can have a RollingFileListeners and tell these objects about
important events and pass on config and current state to them. The
listeners can then do further processing, there might be a need to tell
the rolling appender of action taken like files were moved or deleted.
The appender does not need to know where the files are gone (to hadoop
or just zipped for long term storage) but it might help with its
internal state being more consistent with changes to the folder.
On 3/29/2013 9:48 AM, Pranav Bhole wrote:
> Hello to All,
> As per our discussion I am attaching the "Blueprint of Simple,
> Efficient and Reliable HadoopAppender in Log4j". I would love see the
> suggestions.
>
> Thank you
>
>
> On Fri, Mar 22, 2013 at 8:42 PM, Gary Gregory <garydgregory@gmail.com
> <mailto:garydgregory@gmail.com>> wrote:
>
> An AppScale appender might be interesting:
> http://appscale.cs.ucsb.edu/datastores.html
>
> Gary
>
>
> On Fri, Mar 22, 2013 at 4:34 PM, Christian Grobmeier
> <grobmeier@gmail.com <mailto:grobmeier@gmail.com>> wrote:
>
> +1
>
> I would love to support a GSOC student, and if your more concrete
> proposal meets some interest here I am willing to actually help.
> That said, while Apache Flume is great, its maybe a bit "too
> much". I
> already have had some thoughts on some kind of a server which
> utilizes
> receivers to send data to $x. Less features than Flume, but
> easy to
> setup. Not sure if that has some value.
>
> Pranav, please let us hear more of your ideas.
>
>
>
> On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers
> <ralph.goers@dslextreme.com
> <mailto:ralph.goers@dslextreme.com>> wrote:
> > The Flume Appender leverages Apache Flume to route data into
> various places.
> > The primary sponsor of Flume is Cloudera, so naturally Flume
> supports
> > writing data into Hadoop. In addition, my employer is using
> Flume to write
> > data into Cassandra. That said, we would welcome
> contributions and if you
> > can provide more details on how you would implement your
> idea I'd love to
> > see them. Perhaps you can create a page on the logging wiki
> with your
> > proposal.
> >
> > Ralph
> >
> > On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
> >
> > Hello to all,
> > This is Pranav Bhole, I am Master student at The
> University of Texas
> > at Dallas. My research interest is Big Data. I haven been
> using Log4j
> > extensively as core since 5-6 years in my academic and
> professional work.
> > Recently an idea came up in my mind by facing some of the
> difficulties in
> > managing the TeraBytes of Log files. I would like to
> implement this idea as
> > plug in or functionality in the existing log4j appender
> module as student of
> > Google Summer of Code 2013.
> >
> > Short description of the idea:
> > Server appends the bulk of log files and in the most cases
> server lacks with
> > the storage space for these logs files and also computing on
> such bulk of
> > file is costly for the server. With the consideration of
> this problem, idea
> > proposes to write a module which could be able to move these
> files into
> > Public (S3 of AWS, Azure) or private cloud (Hadoop) on the
> rolling basis
> > based on the configuration file. To resolve the computing
> layer objective,
> > the idea proposes the Big Data Query generator based on the
> logging format
> > used. Such Big Data Queries will include MapReduce, PIG etc.
> Administrator
> > would be able to run these BigData queries generated by
> Log4j to track the
> > keywords in the logs like error number, TimeStamp or any
> other arbitrary
> > string.
> >
> > I would like to appreciate to all of you for reading this
> idea. I would
> > really love to get involved in Log4j development team with
> your support and
> > suggestion on this idea.
> >
> > Thank you very much.
> >
> > --
> > Pranav Bhole
> > Student of MS in Computer Science for Fall 2012,
> > University of Texas at Dallas
> > http://www.linkedin.com/in/pranavbhole
> > Cell Phone No: 972-978-6108 <tel:972-978-6108>.
> >
> >
>
>
>
> --
> http://www.grobmeier.de
> https://www.timeandbill.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> log4j-dev-unsubscribe@logging.apache.org
> <mailto:log4j-dev-unsubscribe@logging.apache.org>
> For additional commands, e-mail:
> log4j-dev-help@logging.apache.org
> <mailto:log4j-dev-help@logging.apache.org>
>
>
>
>
> --
> E-Mail: garydgregory@gmail.com <mailto:garydgregory@gmail.com> |
> ggregory@apache.org <mailto:ggregory@apache.org>
> JUnit in Action, 2nd Ed: http://bit.ly/ECvg0
> Spring Batch in Action: http://bit.ly/bqpbCK
> Blog: http://garygregory.wordpress.com
> <http://garygregory.wordpress.com/>
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>
>
>
>
> --
> Pranav Bhole
> Student of MS in Computer Science for Fall 2012,
> University of Texas at Dallas
> http://www.linkedin.com/in/pranavbhole
> Cell Phone No: 972-978-6108.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-dev-help@logging.apache.org
[Attachment #3 (text/html)]
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">From what I could see you plan to edit
the rolling file appender so it can take a custom action on some
events (roll over when already rolled over files is > X) at
which point files are pushed to another system. Rolling file
appender would continue to do what it does natively. I think a
better design would be to enhance the appender in a generic way so
it can have a RollingFileListeners and tell these objects about
important events and pass on config and current state to them. The
listeners can then do further processing, there might be a need to
tell the rolling appender of action taken like files were moved or
deleted. The appender does not need to know where the files are
gone (to hadoop or just zipped for long term storage) but it might
help with its internal state being more consistent with changes to
the folder.<br>
<br>
<br>
On 3/29/2013 9:48 AM, Pranav Bhole wrote:<br>
</div>
<blockquote
cite="mid:CANvspFBpZ-+Aj7SuxEyTRoTk2mgJeBiB+P5A4xga0qo5mrHOFQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hello to All,
<div style="">As per our discussion I am attaching the
"Blueprint of Simple, Efficient and Reliable HadoopAppender in
Log4j". I would love see the suggestions.<br>
<br>
Thank you</div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On Fri, Mar 22, 2013 at 8:42 PM, Gary
Gregory <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:garydgregory@gmail.com" \
target="_blank">garydgregory@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>An AppScale appender might be interesting: <a
moz-do-not-send="true"
href="http://appscale.cs.ucsb.edu/datastores.html"
rel="nofollow" \
target="_blank">http://appscale.cs.ucsb.edu/datastores.html</a><br> <br>
</div>
Gary<br>
</div>
<div class="gmail_extra">
<div>
<div class="h5">
<br>
<br>
<div class="gmail_quote">On Fri, Mar 22, 2013 at
4:34 PM, Christian Grobmeier <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:grobmeier@gmail.com"
target="_blank">grobmeier@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
+1<br>
<br>
I would love to support a GSOC student, and if
your more concrete<br>
proposal meets some interest here I am willing
to actually help.<br>
That said, while Apache Flume is great, its
maybe a bit "too much". I<br>
already have had some thoughts on some kind of a
server which utilizes<br>
receivers to send data to $x. Less features than
Flume, but easy to<br>
setup. Not sure if that has some value.<br>
<br>
Pranav, please let us hear more of your ideas.<br>
<div>
<div><br>
<br>
<br>
On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers
<<a moz-do-not-send="true"
href="mailto:ralph.goers@dslextreme.com"
target="_blank">ralph.goers@dslextreme.com</a>>
wrote:<br>
> The Flume Appender leverages Apache
Flume to route data into various places.<br>
> The primary sponsor of Flume is
Cloudera, so naturally Flume supports<br>
> writing data into Hadoop. In addition,
my employer is using Flume to write<br>
> data into Cassandra. That said, we
would welcome contributions and if you<br>
> can provide more details on how you
would implement your idea I'd love to<br>
> see them. Perhaps you can create a
page on the logging wiki with your<br>
> proposal.<br>
><br>
> Ralph<br>
><br>
> On Mar 22, 2013, at 2:04 AM, Pranav
Bhole wrote:<br>
><br>
> Hello to all,<br>
> This is Pranav Bhole, I \
am Master student at The University of Texas<br>
> at Dallas. My research interest is Big
Data. I haven been using Log4j<br>
> extensively as core since 5-6 years in
my academic and professional work.<br>
> Recently an idea came up in my mind by
facing some of the difficulties in<br>
> managing the TeraBytes of Log files. I
would like to implement this idea as<br>
> plug in or functionality in the
existing log4j appender module as student of<br>
> Google Summer of Code 2013.<br>
><br>
> Short description of the idea:<br>
> Server appends the bulk of log files
and in the most cases server lacks with<br>
> the storage space for these logs files
and also computing on such bulk of<br>
> file is costly for the server. With the
consideration of this problem, idea<br>
> proposes to write a module which could
be able to move these files into<br>
> Public (S3 of AWS, Azure) or private
cloud (Hadoop) on the rolling basis<br>
> based on the configuration file. To
resolve the computing layer objective,<br>
> the idea proposes the Big Data Query
generator based on the logging format<br>
> used. Such Big Data Queries will
include MapReduce, PIG etc. Administrator<br>
> would be able to run these BigData
queries generated by Log4j to track the<br>
> keywords in the logs like error number,
TimeStamp or any other arbitrary<br>
> string.<br>
><br>
> I would like to appreciate to all of
you for reading this idea. I would<br>
> really love to get involved in Log4j
development team with your support and<br>
> suggestion on this idea.<br>
><br>
> Thank you very much.<br>
><br>
> --<br>
> Pranav Bhole<br>
> Student of MS in Computer Science for
Fall 2012,<br>
> University of Texas at Dallas<br>
> <a moz-do-not-send="true"
href="http://www.linkedin.com/in/pranavbhole"
\
target="_blank">http://www.linkedin.com/in/pranavbhole</a><br> > Cell Phone No: \
<a moz-do-not-send="true"
href="tel:972-978-6108"
value="+19729786108" \
target="_blank">972-978-6108</a>.<br> ><br>
><br>
<br>
<br>
<br>
</div>
</div>
<span><font color="#888888">--<br>
<a moz-do-not-send="true"
href="http://www.grobmeier.de"
target="_blank">http://www.grobmeier.de</a><br>
<a moz-do-not-send="true"
href="https://www.timeandbill.de"
target="_blank">https://www.timeandbill.de</a><br>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a
moz-do-not-send="true"
href="mailto:log4j-dev-unsubscribe@logging.apache.org"
\
target="_blank">log4j-dev-unsubscribe@logging.apache.org</a><br> For additional \
commands, e-mail: <a moz-do-not-send="true"
href="mailto:log4j-dev-help@logging.apache.org"
\
target="_blank">log4j-dev-help@logging.apache.org</a><br> <br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
<span class="HOEnZb"><font color="#888888">-- <br>
E-Mail: <a moz-do-not-send="true"
href="mailto:garydgregory@gmail.com"
target="_blank">garydgregory@gmail.com</a> | <a
moz-do-not-send="true"
href="mailto:ggregory@apache.org" \
target="_blank">ggregory@apache.org </a><br>
JUnit in Action, 2nd Ed: <a moz-do-not-send="true"
href="http://bit.ly/ECvg0" \
target="_blank">http://bit.ly/ECvg0</a><br> Spring Batch in Action: <a \
moz-do-not-send="true" href="http://bit.ly/bqpbCK" rel="nofollow"
target="_blank">http://bit.ly/bqpbCK</a><br>
Blog: <a moz-do-not-send="true"
href="http://garygregory.wordpress.com/"
target="_blank">http://garygregory.wordpress.com</a>
<br>
Home: <a moz-do-not-send="true"
href="http://garygregory.com/" \
target="_blank">http://garygregory.com/</a><br> Tweet! <a moz-do-not-send="true"
href="http://twitter.com/GaryGregory"
target="_blank">http://twitter.com/GaryGregory</a>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Pranav Bhole<br>
Student of MS in Computer Science for Fall 2012,<br>
University of Texas at Dallas<br>
<a moz-do-not-send="true"
href="http://www.linkedin.com/in/pranavbhole"
target="_blank">http://www.linkedin.com/in/pranavbhole</a><br>
Cell Phone No: 972-978-6108.<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">
---------------------------------------------------------------------
To unsubscribe, e-mail: <a class="moz-txt-link-abbreviated" \
href="mailto:log4j-dev-unsubscribe@logging.apache.org">log4j-dev-unsubscribe@logging.apache.org</a>
For additional commands, e-mail: <a class="moz-txt-link-abbreviated" \
href="mailto:log4j-dev-help@logging.apache.org">log4j-dev-help@logging.apache.org</a></pre>
</blockquote>
<br>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic