[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Add hostname when sinking nginx log to kafka
From: Zhun Shen <shenzhunallen () gmail ! com>
Date: 2015-07-19 10:10:39
Message-ID: CAPjEbHm2UPd2j5n=8hxPUZB1GRSOK0oipZK5pkzoMNqfVGNS5Q () mail ! gmail ! com
[Download RAW message or body]
Hello,
I'm using Flume 1.6 to collect nginx log then sink the log to kafka. I want
to add hostname of nginx when I sink the log to kafka in order to analyse
the web traffic of different hosts.
Here is my flume configuration file:
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /data/tmp/cs.log
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = host
a1.sources.r1.interceptors.i1.hostHeader = hostname
a1.sinks = s1
a1.sinks.s1.channel = c1
a1.sinks.s1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.s1.zk.connect = 10.5.2.129:2181,10.5.2.127:2181,10.5.2.131:2181,
10.5.2.130:2181,10.5.2.132:2181
a1.sinks.s1.batchsize = 10
a1.sinks.s1.topic = testflume
a1.sinks.s1.key = test
a1.sinks.s1.brokerList = 10.5.2.112:9092,10.5.2.113:9092
a1.sinks.s1.requiredAcks = -1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
But it did not work, the key was still null. I could not find the hostname
anywhere.
{Event} testflume : null : 83.38.189.80 - - [12/Nov/2014:05:05:02 -0800]
"GET /es/index.php?main_page=checkout_login HTTP/1.0" 302 0 "
http://www.lightinthebox.com/es/index.php?main_page=shopping_cart"
"Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/38.0.2125.111 Safari/537.36"
If you know how to solve it, let me know.
Many thanks.
--
Shen Zhun (Allen)
Data Mining at LightnInTheBox.com
Email: shenzhunallen@gmail.com
GitHub: https://github.com/shenzhun
LinkedIn: http://www.linkedin.com/in/shenzhun
[Attachment #3 (text/html)]
<div dir="ltr">Hello,<div>I'm using Flume 1.6 to collect nginx log then sink the \
log to kafka. I want to add hostname of nginx when I sink the log to kafka in order \
to analyse the web traffic of different hosts.</div><div><br></div><div>Here is my \
flume configuration file:</div><div><div>a1.sources = r1</div><div>a1.channels = \
c1</div><div><br></div><div>a1.sources.r1.type = \
exec</div><div>a1.sources.r1.channels = c1</div><div>a1.sources.r1.command = tail -F \
/data/tmp/cs.log</div><div>a1.sources.r1.interceptors = \
i1</div><div>a1.sources.r1.interceptors.i1.type = \
host</div><div>a1.sources.r1.interceptors.i1.hostHeader = \
hostname</div><div><br></div><div>a1.sinks = s1</div><div>a1.sinks.s1.channel = \
c1</div><div>a1.sinks.s1.type = \
org.apache.flume.sink.kafka.KafkaSink</div><div>a1.sinks.s1.zk.connect = <a \
href="http://10.5.2.129:2181">10.5.2.129:2181</a>,<a \
href="http://10.5.2.127:2181">10.5.2.127:2181</a>,<a \
href="http://10.5.2.131:2181">10.5.2.131:2181</a>,<a \
href="http://10.5.2.130:2181">10.5.2.130:2181</a>,<a \
href="http://10.5.2.132:2181">10.5.2.132:2181</a></div><div>a1.sinks.s1.batchsize = \
10</div><div>a1.sinks.s1.topic = testflume</div><div>a1.sinks.s1.key = \
test</div><div>a1.sinks.s1.brokerList = <a \
href="http://10.5.2.112:9092">10.5.2.112:9092</a>,<a \
href="http://10.5.2.113:9092">10.5.2.113:9092</a></div><div>a1.sinks.s1.requiredAcks \
= -1</div><div><br></div><div>a1.channels.c1.type = \
memory</div><div>a1.channels.c1.capacity = \
1000</div><div>a1.channels.c1.transactionCapacity = \
100</div></div><div><br></div><div>But it did not work, the key was still null. I \
could not find the hostname anywhere.</div><div><br></div><div>{Event} testflume : \
null : 83.38.189.80 - - [12/Nov/2014:05:05:02 -0800] "GET \
/es/index.php?main_page=checkout_login HTTP/1.0" 302 0 "<a \
href="http://www.lightinthebox.com/es/index.php?main_page=shopping_cart">http://www.lightinthebox.com/es/index.php?main_page=shopping_cart</a>" \
"Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) \
Chrome/38.0.2125.111 Safari/537.36" </div><div><br></div><div>If you know how \
to solve it, let me know.</div><div><br></div><div>Many \
thanks.</div><div><br><div><br></div><div><div><div class="gmail_signature"><div \
dir="ltr"><div class="gmail_signature"><div dir="ltr"><div \
style="color:rgb(0,0,0);font-family:HelveticaNeue,'Helvetica \
Neue',Helvetica,Arial,'Lucida Grande',sans-serif"><span \
style="font-family:'Courier \
New',courier,monaco,monospace,sans-serif">--</span></div><div \
style="color:rgb(0,0,0);font-family:HelveticaNeue,'Helvetica \
Neue',Helvetica,Arial,'Lucida Grande',sans-serif"><span \
style="font-family:'Courier New',courier,monaco,monospace,sans-serif">Shen \
Zhun (Allen)</span></div><div \
style="color:rgb(0,0,0);font-family:HelveticaNeue,'Helvetica \
Neue',Helvetica,Arial,'Lucida Grande',sans-serif"><span \
style="font-family:'Courier New',courier,monaco,monospace,sans-serif">Data \
Mining at LightnInTheBox.com</span></div><div \
style="color:rgb(0,0,0);font-family:'Courier \
New',courier,monaco,monospace,sans-serif">Email: <a \
href="mailto:shenzhunallen@gmail.com" \
target="_blank">shenzhunallen@gmail.com</a></div><div \
style="color:rgb(0,0,0);font-family:'Courier \
New',courier,monaco,monospace,sans-serif">GitHub: <a rel="nofollow" \
href="https://github.com/shenzhun" \
style="margin:0px;padding:0px;color:blue;outline:none;line-height:1.22;background-color:rgba(0,0,0,0)" \
target="_blank">https://github.com/shenzhun</a></div><div \
style="color:rgb(0,0,0);font-family:'Courier \
New',courier,monaco,monospace,sans-serif">LinkedIn: <a rel="nofollow" \
href="http://www.linkedin.com/in/shenzhun" title="View public profile" \
name="UNIQUE_ID_SafeHtmlFilter_webProfileURL" \
style="margin:0px;padding:0px;color:blue;outline:none;background-color:rgba(0,0,0,0)" \
target="_blank">http://www.linkedin.com/in/shenzhun</a></div></div></div></div></div></div>
</div></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic