[prev in list] [next in list] [prev in thread] [next in thread]
List: bro
Subject: Re: [Bro] logs-to-elasticsearch.bro error
From: Daniel Guerra <daniel.guerra69 () gmail ! com>
Date: 2016-03-25 9:50:56
Message-ID: 7A63317E-EAC2-4612-867F-E01FB67736DD () gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi,
To make this work you need some patches
or use an elasticsearch version lower than 2 (1.7)
I made a docker image for this
https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ \
<https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/> In the git there is \
a map bro-patch https://github.com/danielguerra69/bro-debian-elasticsearch.git \
<https://github.com/danielguerra69/bro-debian-elasticsearch.git>
Regards,
Daniel
> On 25 Mar 2016, at 10:42, mz <mz89924@126.com> wrote:
>
> Dear
> Use logs-to-elasticsearch.bro send logs to ES. Is now work.
>
> ES error logs:
> [2016-03-25 17:30:52,957][DEBUG][action.bulk ] [node-1] \
> [whbro-201603251500][1] failed to execute bulk item (index) index \
> {[whbro-201603251500][dns][AVOtHLQHooGOx5uLgLSQ], \
> source[{"_timestamp":1458898236411,"ts":1458898206267,"uid":"ClbNI74bIcRQ8Gs6Wc","id \
> .orig_h":"10.100.78.88","id.orig_p":137,"id.resp_h":"10.100.79.255","id.resp_p":137, \
> "proto":"udp","trans_id":47282,"query":"ISATAP","qclass":1,"qclass_name":"C_INTERNET \
> ","qtype":32,"qtype_name":"NB","AA":false,"TC":false,"RD":true,"RA":false,"Z":1,"rejected":false}]}
> MapperParsingException[Field [_timestamp] is a metadata field and cannot be added \
> inside a document. Use the index API request parameters.] at \
> org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:213) \
> at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:131)
> at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79)
> at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304)
> at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:500)
> at org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:481)
> at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214)
> at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)
> at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
> at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
> at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
> at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:595)
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263)
> at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260)
> at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Bro config file:
> /usr/local/bro/lib/bro/plugins/Bro_ElasticSearch/scripts/init.bro
> module LogElasticSearch;
>
> export {
> ## Destination for the ES logs. Valid options are
> ## "direct" to directly connect to ES and "nsq" to
> ## transfer the logs into an nsqd instance.
> const destination = "direct" &redef;
>
> ## Name of the ES cluster.
> const cluster_name = "my-application" &redef;
>
> ## ES server.
> const server_host = "10.100.79.10" &redef;
>
> ## ES port.
> const server_port = 9200 &redef;
>
> ## Name of the ES index.
> const index_prefix = "testooo" &redef;
>
> ## Should the index names be in UTC or in local time?
> ## Setting this to true would be more compatible with Kibana and other tools.
> const index_name_in_utc = F &redef;
>
> ## Format for the index names.
> ## Setting this to "%Y.%m.%d-%H" would be more compatible Kibana and other tools.
> #const index_name_fmt = "%Y%m%d" &redef;
> const index_name_fmt = "%Y%m%d%H%M" &redef;
> ## The ES type prefix comes before the name of the related log.
> ## e.g. prefix = "bro\_" would create types of bro_dns, bro_software, etc.
> const type_prefix = "" &redef;
>
> ## The time before an ElasticSearch transfer will timeout. Note that
> ## the fractional part of the timeout will be ignored. In particular,
> ## time specifications less than a second result in a timeout value of
> ## 0, which means "no timeout."
> const transfer_timeout = 2secs;
>
> ## The batch size is the number of messages that will be queued up before
> ## they are sent to be bulk indexed.
> const max_batch_size = 1000 &redef;
>
> ## The maximum amount of wall-clock time that is allowed to pass without
> ## finishing a bulk log send. This represents the maximum delay you
> ## would like to have with your logs before they are sent to ElasticSearch.
> const max_batch_interval = 1min &redef;
>
> ## The maximum byte size for a buffered JSON string to send to the bulk
> ## insert API.
> const max_byte_size = 1024 * 1024 &redef;
>
> ## If the "nsq" destination is given, this is the topic
> ## that Bro will push logs into.
> const nsq_topic = "bro_logs" &redef;
> }
>
> _______________________________________________
> Bro mailing list
> bro@bro-ids.org <mailto:bro@bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro \
> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
[Attachment #5 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br \
class=""></div><div class="">To make this work you need some patches </div><div \
class="">or use an elasticsearch version lower than 2 (1.7)</div><div class=""><br \
class=""></div><div class="">I made a docker image for this</div><div class=""><a \
href="https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/" \
class="">https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/</a></div><div \
class="">In the git there is a map bro-patch </div><div class=""><a \
href="https://github.com/danielguerra69/bro-debian-elasticsearch.git" \
class="">https://github.com/danielguerra69/bro-debian-elasticsearch.git</a></div><div \
class=""><br class=""></div><div class="">Regards,</div><div class=""><br \
class=""></div><div class="">Daniel</div><div class=""><br class=""><div><blockquote \
type="cite" class=""><div class="">On 25 Mar 2016, at 10:42, mz <<a \
href="mailto:mz89924@126.com" class="">mz89924@126.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><div class=""><div class="WordSection1" \
style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: \
normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; \
orphans: auto; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: \
0px;"><div style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; \
font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Dear<o:p \
class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><span \
lang="EN-US" class=""> Use \
logs-to-elasticsearch.bro send logs to ES. Is now work.<o:p \
class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><span \
lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm \
0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: Calibri, \
sans-serif;" class=""><b class=""><span lang="EN-US" class="">ES error logs:<o:p \
class=""></o:p></span></b></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
[2016-03-25 17:30:52,957][DEBUG][action.bulk \
] [node-1] [whbro-201603251500][1] failed to execute bulk item (index) index \
{[whbro-201603251500][dns][AVOtHLQHooGOx5uLgLSQ], \
source[{"_timestamp":1458898236411,"ts":1458898206267,"uid":"ClbNI74bIcRQ8Gs6Wc","id.o \
rig_h":"10.100.78.88","id.orig_p":137,"id.resp_h":"10.100.79.255","id.resp_p":137,"pro \
to":"udp","trans_id":47282,"query":"ISATAP","qclass":1,"qclass_name":"C_INTERNET","qty \
pe":32,"qtype_name":"NB","AA":false,"TC":false,"RD":true,"RA":false,"Z":1,"rejected":false}]}<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class="">MapperParsingException[Field [_timestamp] is a \
metadata field and cannot be added inside a document. Use the index API request \
parameters.]<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm \
0.0001pt; text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><i class=""><span lang="EN-US" \
class=""> at \
org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:213)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:131)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:500)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:481)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> at \
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:595)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> \
at java.lang.Thread.run(Thread.java:745)<o:p class=""></o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" class=""><o:p \
class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><b class=""><span lang="EN-US" class="">Bro config file:<o:p \
class=""></o:p></span></b></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><b \
class=""><span lang="EN-US" \
class="">/usr/local/bro/lib/bro/plugins/Bro_ElasticSearch/scripts/init.bro<o:p \
class=""></o:p></span></b></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class="">module LogElasticSearch;<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" class="">export {<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> ## \
Destination for the ES logs. Valid options are<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> ## \
"direct" to directly connect to ES and "nsq" to<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> ## \
transfer the logs into an nsqd instance.<o:p class=""></o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" \
class=""> const destination = "direct" \
&redef;<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><i class=""><span lang="EN-US" class=""><o:p \
class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><i class=""><span lang="EN-US" \
class=""> ## Name of the ES cluster.<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> const \
cluster_name = "my-application" &redef;<o:p class=""></o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" class=""><o:p \
class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><i class=""><span lang="EN-US" \
class=""> ## ES server.<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> const \
server_host = "10.100.79.10" &redef;<o:p class=""></o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" class=""><o:p \
class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><i class=""><span lang="EN-US" \
class=""> ## ES port.<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> const \
server_port = 9200 &redef;<o:p class=""></o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" class=""><o:p \
class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
text-align: justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" \
class=""><i class=""><span lang="EN-US" \
class=""> ## Name of the ES index.<o:p \
class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; text-align: \
justify; font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><i \
class=""><span lang="EN-US" class=""> const \
index_prefix = "testooo" &redef;<o:p class=""></o:p></span></i></div><div \
style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-size: 10.5pt; font-family: \
Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" class=""><o:p \
class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; \
_______________________________________________
Bro mailing list
bro@bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic