[prev in list] [next in list] [prev in thread] [next in thread] 

List:       flume-user
Subject:    Re: Realtime sensor's tcpip data to hadoop
From:       Bertrand Dechoux <dechouxb () gmail ! com>
Date:       2014-05-09 13:46:24
Message-ID: CAO6W-2dWX4zGkV9pSUySfE_TdtpDomTAWs7U9QcCy+O_kbEjhw () mail ! gmail ! com
[Download RAW message or body]

Flume is indeed something you should look into. 'Log files' is a
simplification. Flume really handles events and yes logs are a common kind
of event but not the only one.

Regards

Bertrand

On Wed, May 7, 2014 at 4:48 AM, Alex Lee <eliyart@hotmail.com> wrote:

> Sensors' may send tcpip data to server. Each sensor may send tcpip data
> like a stream to the server, the quatity of the sensors and the data rate
> of the data is high.
>
> Firstly, how the data from tcpip can be put into hadoop. It need to do
> some process and store in hbase. Does it need through save to data files
> and put into hadoop or can be done in some direct ways from tcpip. Is there
> any software module can take care of this. Searched that Ganglia Nagios and
> Flume may do it. But when looking into details, ganglia and nagios are
> more for monitoring hadoop cluster itself. Flume is for log files.
>
> Secondly, if the total network traffic from sensors are over the limit of
> one lan port, how to share the loads, is there any component in hadoop to
> make this done automatically.
>
> Any suggestions, thanks.
>

[Attachment #3 (text/html)]

<div dir="ltr">Flume is indeed something you should look into. &#39;Log files&#39; is \
a simplification. Flume really handles events and yes logs are a common kind of event \
but not the only one.<div><br></div><div>Regards</div> \
<div><br></div><div>Bertrand</div><div class="gmail_extra"><br><div \
class="gmail_quote">On Wed, May 7, 2014 at 4:48 AM, Alex Lee <span dir="ltr">&lt;<a \
href="mailto:eliyart@hotmail.com" target="_blank">eliyart@hotmail.com</a>&gt;</span> \
wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex">


<div><div dir="ltr">Sensors&#39;  may send tcpip data to server. Each sensor may send \
tcpip  data like a stream to the server, the quatity of the sensors and the data rate \
of the data is high.<br>  <br>Firstly, how the  data from tcpip can be put into \
hadoop. It need to do some process and store in hbase. Does it need through save to  \
data files and put into hadoop  or can be done in some  direct ways from tcpip. Is \
there any software module can take care of this.  Searched that  Ganglia  Nagios and \
Flume may do it. But when looking into details, ganglia and nagios are more  for \
monitoring hadoop cluster itself. Flume is for log files.<br>  <br>Secondly, if  the  \
total network traffic from sensors are over the limit of one lan port, how to share \
the loads, is there any component in hadoop to make this done automatically.<br>  \
<br>Any suggestions, thanks.<br> 		 	   		  </div> </div>
</blockquote></div><br></div></div>



[prev in list] [next in list] [prev in thread] [next in thread]