[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Re: Re: flume hive sink not work
From: iain wright <iainwrig () gmail ! com>
Date: 2015-10-22 6:53:29
Message-ID: CAMhkWdAGgj+Y7C78A-zddg3HEp_1ME9weyV63a-xvEkqwuXrKA () mail ! gmail ! com
[Download RAW message or body]
What is the result from ping bigdata1
Seems like it's not resolving from your telnet test
Try to telnet to the ip of bigdata on the same port.
If that works you can try changing bigdata1 to the IP of the machine in
your flume config (is it the same host flume is running on? If so use
127.0.0.1)
or sort out setting an A record in your dns server for bigdata1, or an
entry in /etc/hosts , etc.
On Oct 21, 2015 11:23 PM, "lizhenmxcz@163.com" <lizhenmxcz@163.com> wrote:
>
> telnet say "nane or service not known",but the port is listening
> tcp 0 0 0.0.0.0:9083 0.0.0.0:
> * LISTEN 32119/java
> ------------------------------
> lizhenmxcz@163.com
>
>
> *From:* iain wright <iainwrig@gmail.com>
> *Date:* 2015-10-22 14:13
> *To:* user <user@flume.apache.org>
> *Subject:* Re: flume hive sink not work
>
> Are you able to telnet to bigdata1 on port 9083?
> On Oct 21, 2015 7:27 PM, "lizhenmxcz@163.com" <lizhenmxcz@163.com> wrote:
>
> >
> > hi all：
> > I use flume to import data from syslog to hive,but encount the follow
> > errors.
> >
> >
> > 2015-10-22 10:05:05,115 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - \
> > org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:324)] k2 : Failed \
> > connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb', \
> > table='dns_request', partitionVals=[] }
> > org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to \
> > EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb', \
> > table='dns_request', partitionVals=[] }
> > at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
> >
> > at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
> >
> > at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
> > at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
> >
> > at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> >
> > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed \
> > connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb', \
> > table='dns_request', partitionVals=[] }
> > at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)
> >
> > at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
> > ... 6 more
> > Caused by: java.util.concurrent.TimeoutException
> > at java.util.concurrent.FutureTask.get(FutureTask.java:201)
> >
> > at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:431)
> >
> > at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:373)
> > ... 7 more
> >
> >
> > my configuration is:
> >
> >
> > a1.sources = r1
> > a1.channels = c1 c2
> > a1.sinks = k1 k2
> >
> > a1.sources.r1.type = syslogudp
> > a1.sources.r1.port = 514
> > a1.sources.r1.host = 192.168.55.246
> >
> > a1.sources.r1.channels = c1 c2
> > a1.sources.r1.interceptors = i1
> > a1.sources.r1.interceptors.i1.type = regex_extractor
> > a1.sources.r1.interceptors.i1.regex = Dns(.*)\\[
> > a1.sources.r1.interceptors.i1.serializers = t1
> > a1.sources.r1.interceptors.i1.serializers.t1.name = type
> >
> > a1.sources.r1.selector.type = multiplexing
> > a1.sources.r1.selector.header = type
> > a1.sources.r1.selector.mapping.Request = c1
> > a1.sources.r1.selector.mapping.Answer = c2
> >
> > a1.sinks.k2.type = hive
> > a1.sinks.k2.channel = c1
> > a1.sinks.k2.hive.metastore = thrift://bigdata1:9083
> > a1.sinks.k2.hive.database = dnsdb
> > a1.sinks.k2.hive.table = dns_request
> > a1.sinks.k2.hive.partiton = %Y,%m,%d,%H
> > a1.sinks.k2.hive.txnsPerBatchAsk = 2
> > a1.sinks.k2.batchSize = 10
> > a1.sinks.k2.serializer = delimited
> > a1.sinks.k2.serializer.delimiter = ,
> > a1.sinks.k2.serializer.fieldnames = timepoint,random,sip,dip,spt,type,name
> >
> > a1.sinks.k1.type = hive
> > a1.sinks.k1.channel = c2
> > a1.sinks.k1.hive.metastore = thrift://bigdata1:9083
> > a1.sinks.k1.hive.database = Dnsdb
> > a1.sinks.k1.hive.table = dns_answer
> > a1.sinks.k1.hive.partiton = %Y,%m,%d,%H
> > a1.sinks.k1.hive.txnsPerBatchAsk = 2
> > a1.sinks.k1.batchSize = 10
> > a1.sinks.k1.serializer = delimited
> > a1.sinks.k1.serializer.delimiter = ,
> >
> > a1.sinks.k1.serializer.fieldnames = \
> > timepoint,random,sip,dip,dpt,name,nosuchname,typemax,typecname,typeaddr,authservername,additionalrecords
> >
> > help me please,thanks.
> >
> > ------------------------------
> > lizhenmxcz@163.com
> >
>
[Attachment #3 (text/html)]
<p dir="ltr">What is the result from ping bigdata1</p>
<p dir="ltr">Seems like it's not resolving from your telnet test</p>
<p dir="ltr">Try to telnet to the ip of bigdata on the same port.</p>
<p dir="ltr">If that works you can try changing bigdata1 to the IP of the machine in \
your flume config (is it the same host flume is running on? If so use 127.0.0.1)</p> \
<p dir="ltr">or sort out setting an A record in your dns server for bigdata1, or an \
entry in /etc/hosts , etc.<br> </p>
<div class="gmail_quote">On Oct 21, 2015 11:23 PM, "<a \
href="mailto:lizhenmxcz@163.com">lizhenmxcz@163.com</a>" <<a \
href="mailto:lizhenmxcz@163.com">lizhenmxcz@163.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div> \
<div><span></span><br></div><div>telnet say "nane or service not known",but \
the port is listening </div> <div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)">tcp 0 \
0 <a href="http://0.0.0.0:9083" target="_blank">0.0.0.0:9083</a> \
0.0.0.0:* LISTEN \
32119/java</span></div><hr style="width:210px;min-height:1px" color="#b5c4df" \
size="1" align="left"> <div><span><div \
style="MARGIN:10px;FONT-FAMILY:verdana;FONT-SIZE:10pt"><div><a \
href="mailto:lizhenmxcz@163.com" \
target="_blank">lizhenmxcz@163.com</a></div></div></span></div> <blockquote \
style="margin-top:0px;margin-bottom:0px;margin-left:0.5em"><div> </div><div \
style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div \
style="PADDING-RIGHT:8px;PADDING-LEFT:8px;FONT-SIZE:12px;FONT-FAMILY:tahoma;COLOR:#000000;BACKGROUND:#efefef;PADDING-BOTTOM:8px;PADDING-TOP:8px"><div><b>From:</b> \
<a href="mailto:iainwrig@gmail.com" target="_blank">iain \
wright</a></div><div><b>Date:</b> 2015-10-22 14:13</div><div><b>To:</b> <a \
href="mailto:user@flume.apache.org" \
target="_blank">user</a></div><div><b>Subject:</b> Re: flume hive sink not \
work</div></div></div><div><div><p dir="ltr">Are you able to telnet to bigdata1 on \
port 9083?</p> <div class="gmail_quote">On Oct 21, 2015 7:27 PM, "<a \
href="mailto:lizhenmxcz@163.com" target="_blank">lizhenmxcz@163.com</a>" <<a \
href="mailto:lizhenmxcz@163.com" target="_blank">lizhenmxcz@163.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>
<div><span></span><br></div>
<div><span style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)">hi \
all：</span></div><div>I use flume to import data from syslog to hive,but encount \
the follow <span style="font-size:12pt;line-height:1.5;background-color:window">errors.</span><span \
style="font-size:12pt;line-height:1.5;background-color:window"> \
</span></div><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"><br></span></div><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)">2015-10-22 10:05:05,115 \
(SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - \
org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:324)] k2 : Failed \
connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083', \
database='dnsdb', table='dns_request', partitionVals=[] \
}<br>org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to \
EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb', \
table='dns_request', partitionVals=[] }<br> at \
org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)<br> \
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)<br> \
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)<br> \
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)<br> \
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)<br> \
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)<br> \
at java.lang.Thread.run(Thread.java:745)<br>Caused by: \
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to \
EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb', \
table='dns_request', partitionVals=[] }<br> at \
org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)<br> \
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)<br> \
... 6 more<br>Caused by: java.util.concurrent.TimeoutException<br> \
at java.util.concurrent.FutureTask.get(FutureTask.java:201)<br> at \
org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:431)<br> \
at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:373)<br> \
... 7 more</span></div><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"><br></span></div><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"><br></span></div><div>my \
configuration is:</div><div><br></div><div><br></div><div><span \
style="background-color:rgba(0,0,0,0);font-size:12pt;line-height:1.5">a1.sources = \
r1</span></div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)">a1.channels = c1 \
c2<br>a1.sinks = k1 k2<br><br>a1.sources.r1.type = \
syslogudp<br>a1.sources.r1.port = 514<br>a1.sources.r1.host = \
192.168.55.246<br><br>a1.sources.r1.channels = c1 c2<br>a1.sources.r1.interceptors \
= i1<br>a1.sources.r1.interceptors.i1.type = \
regex_extractor<br>a1.sources.r1.interceptors.i1.regex = \
Dns(.*)\\[<br>a1.sources.r1.interceptors.i1.serializers = t1<br><a \
href="http://a1.sources.r1.interceptors.i1.serializers.t1.name" \
target="_blank">a1.sources.r1.interceptors.i1.serializers.t1.name</a> = \
type<br><br>a1.sources.r1.selector.type = \
multiplexing<br>a1.sources.r1.selector.header = \
type<br>a1.sources.r1.selector.mapping.Request = \
c1<br>a1.sources.r1.selector.mapping.Answer = c2<br><br>a1.sinks.k2.type = \
hive<br>a1.sinks.k2.channel = c1<br>a1.sinks.k2.hive.metastore = \
thrift://bigdata1:9083<br>a1.sinks.k2.hive.database = \
dnsdb<br>a1.sinks.k2.hive.table = dns_request<br>a1.sinks.k2.hive.partiton = \
%Y,%m,%d,%H<br>a1.sinks.k2.hive.txnsPerBatchAsk = 2<br>a1.sinks.k2.batchSize = \
10<br>a1.sinks.k2.serializer = delimited<br>a1.sinks.k2.serializer.delimiter = \
,<br>a1.sinks.k2.serializer.fieldnames = \
timepoint,random,sip,dip,spt,type,name</span><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"><br></span><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)">a1.sinks.k1.type = \
hive<br>a1.sinks.k1.channel = c2<br>a1.sinks.k1.hive.metastore = \
thrift://bigdata1:9083<br>a1.sinks.k1.hive.database = \
Dnsdb<br>a1.sinks.k1.hive.table = dns_answer<br>a1.sinks.k1.hive.partiton = \
%Y,%m,%d,%H<br>a1.sinks.k1.hive.txnsPerBatchAsk = 2<br>a1.sinks.k1.batchSize = \
10<br>a1.sinks.k1.serializer = delimited<br>a1.sinks.k1.serializer.delimiter = \
,<br>a1.sinks.k1.serializer.fieldnames = \
timepoint,random,sip,dip,dpt,name,nosuchname,typemax,typecname,typeaddr,authservername,additionalrecords<br></span></div><div><span \
style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"><br></span></div><div>help me \
please,thanks.</div><div><br></div><hr style="width:210px;min-height:1px" \
color="#b5c4df" size="1" align="left"> <div><span><div \
style="MARGIN:10px;FONT-FAMILY:verdana;FONT-SIZE:10pt"><div><a \
href="mailto:lizhenmxcz@163.com" \
target="_blank">lizhenmxcz@163.com</a></div></div></span></div> \
</div></div></blockquote></div> </div></div></blockquote>
</div></blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic