[prev in list] [next in list] [prev in thread] [next in thread]
List: ossec-list
Subject: Re: [ossec-list] Re: reindexing logs
From: Jose Luis Ruiz <jose () wazuh ! com>
Date: 2016-09-30 16:07:20
Message-ID: CAORR07YDSj9CJhAYB8Gi22Dp0ve4BVcnfPgPkhbH15fqJNstLA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Roberto, nice news :)
Please feel free to send pull request to Wazuh and Ossec with your
improvements and new rules, the Ossec community will appreciate.
Regards
-----------------------
Jose Luis Ruiz
Wazuh Inc.
jose@wazuh.com
On September 30, 2016 at 9:00:32 AM,
roberto.mendonca@phoebustecnologia.com.br (
roberto.mendonca@phoebustecnologia.com.br) wrote:
Hi Jose!
The script worked beautifully! rsrs
Very thanks!
Out of this topic, I'm thinking of improving the rules for some
Windows security
events. I do not know if there is already a topic or work on it.
For the ossec generate alerts, for example, the login types:
And then would release on github. I would like to contribute, if possible.
Em quinta-feira, 29 de setembro de 2016 09:53:05 UTC-3, jose escreveu:
>
> Hi Roberto,
>
> About your osseccall you wrote this in the mail
>
> But the file "template =>" /etc/logstash/elastic-ossec-template2.json "I modified \
> the lines 3 and 8. Line 3: from "template", "ossec *" to "template", "ossecall *"
> Line 8: from "ossec": to "ossecall":
>
> You have an space between ossec, ossecall and the wildcard?, if you have,
> you should not. And with the curl procedure:
>
> $ Cd ~ / ossec_tmp / ossec-wazuh / extensions / ElasticSearch / && curl -XPUT \
> "http: // localhost: 9200 / _template / ossec /" -d "@ elastic-ossec-template.json" \
> You need to apply the templates for both index.
>
> For your last question, in this mail you have a bash script to reindex the
> index. Please use carefully and check with curl
> 'localhost:9200/_cat/indices?v' after every step that the script is doing
> well.
>
> This script has 4 steps:
>
> 1. We move all index without mapping applied to a backup index, we do
> that with the option reindex to apply the new template.
> 2. After the reindex is has finished we can delete the old index.
> 3. Now we can move the backup index to the original name.
> 4. When the step 3 has finished we can delete the backup index.
>
> Pleas take a look the lines 72, 73 and 76, 77 in order to change the index
> name from ossec-$index_elastic_name and ossec-$index_elastic_name by
> ossecall-$index_elastic_name and ossecall-$index_elastic_name because
> probably you need to run this script for your two index.
>
> This one of a few utils that wazuh will release soon.
>
> #!/bin/bash
>
> # Copyright (C) 2015-2016 Wazuh, Inc.All rights reserved.
> # Wazuh.com
> #
> # This program is a free software; you can redistribute it
> # and/or modify it under the terms of the GNU General Public
> # License (version 2) as published by the FSF - Free Software
> # Foundation.
>
> # Elasticsearch Reindexing
> # Requires:
> # Elasticsearch 2.3 or superior
>
> if [ $# -ne 4 ]
> then
> echo "Usage: ./wazuh_elastic_reindex_index.sh date_from date_to elasticsearch_ip \
> step" echo -e "\tDate format: YYYY-MM-DD"
> echo -e "\tStep: 1|2|3|4"
> echo -e "\tExample: ./wazuh_elastic_reindex_index.sh 20160826 20160901 10.0.0.20 1"
> echo -e "\tNote: Each step takes its time to perform the actions required. Review: \
> tail -f /var/log/elasticsearch/ossec.log" exit 0
> fi
>
> ## Arguments
> FROM=$1
> TO=$2
> ELASTIC_IP=$3
> STEP=$4
>
> ## Main
> startdate=$(date -d $FROM +"%Y%m%d")
> enddate=$(date -d $TO +"%Y%m%d")
>
> if [ $startdate -ge $enddate ];
> then
> echo "The date_from $startdate is bigger than date_to $enddate, please review this \
> arguments"; exit 1
> fi
>
> startdate=$(date -I -d "$FROM") || exit -1
> enddate=$(date -I -d "$TO") || exit -1
>
> echo -e "\n### Start reindexing [STEP $STEP], from $startdate to $enddate are you \
> sure? please confirm with YES/NO?" read ADDRANSWER
>
> exist_index () {
> request="$ELASTIC_IP:9200/$1"
> exist=`curl -s -XHEAD -i $request | head -n 1 | cut -d' ' -f2`
> }
>
> reindex () {
> request="$ELASTIC_IP:9200/_reindex"
> request_body='{ "source": { "index": "'"$1"'" }, "dest": { "index": "'"$2"'" }}'
> curl_result=`curl -s -XPOST $request -d "$request_body"`
> echo $curl_result
> }
>
> delete_index () {
> request="$ELASTIC_IP:9200/$1"
> curl_result=`curl -s -XDELETE $request`
> echo $curl_result
> }
>
> if [ $ADDRANSWER == 'YES' ]
> then
> d="$FROM"
> while [ "$d" != "$enddate" ]; do
> index_elastic_name=` echo $d | sed 's/-/\./g'`
>
> if [ $STEP == '1' ] || [ $STEP == '2' ]; then
> src_index="ossec-$index_elastic_name"
> dst_index="ossec-$index_elastic_name-b"
> exist_index $src_index
> elif [ $STEP == '3' ] || [ $STEP == '4' ]; then
> src_index="ossec-$index_elastic_name-b"
> dst_index="ossec-$index_elastic_name"
> exist_index $src_index
> else
> echo "Bad argument: step: $STEP"
> exit 1
> fi
>
> if [ $exist != '404' ]; then
> if [ $STEP == '1' ]; then
> echo "### 1. Reindexing: $src_index -> $dst_index"
> reindex $src_index $dst_index
> elif [ $STEP == '2' ]; then
> echo "### 2. Deleting old index: $src_index"
> delete_index $src_index
> elif [ $STEP == '3' ]; then
> echo "### 3. Reindexing: $src_index"
> reindex $src_index $dst_index
> elif [ $STEP == '4' ]; then
> echo "### 4. Deleting intemediate index: $src_index"
> delete_index $src_index
> fi
> else
> echo "### Index $src_index doest not exist. Skipping."
> fi
>
> # Update date.
> d=$(date -I -d "$d + 1 day")
> done
>
> echo -e "\nPlease check 'curl -XGET ${ELASTIC_IP}:9200/_cat/indices' to re-check \
> the indices" echo "Reindexing ended [STEP $STEP]."
> else
> echo "This script is finished because you don't confirm with YES"
> fi
>
> i hope this helps.
>
> Regards
> -----------------------
> Jose Luis Ruiz
> Wazuh Inc.
> jo...@wazuh.com <javascript:>
>
> On September 29, 2016 at 7:25:09 AM, roberto....@phoebustecnologia.com.br
> <javascript:> (roberto....@phoebustecnologia.com.br <javascript:>) wrote:
>
> Hi Jose, thanks for reply!
>
> Indeed, today the index is in template format. But only ossec index, the
> index ossecall did not work, the fields still appear as "Analyzed Field".
>
> I did not do the procedure:
> $ Cd ~ / ossec_tmp / ossec-wazuh / extensions / ElasticSearch / && curl
> -XPUT "http: // localhost: 9200 / _template / ossec /" -d "@ elastic-ossec
> -template.json"
>
> Just put the logstash output that I said.
>
> But the file "template =>" /etc/logstash/elastic-ossec-template*2*.json "I
> modified the lines 3 and 8.
> Line 3: *from* "template", "ossec *" *to* "template", "ossecall *"
> Line 8: *from* "ossec": *to* "ossecall":
>
> I do not know if it was really necessary to do this. I did this because I
> decided to create a separate index for logs archives.json file. Where
> ossec are logging all.
>
> About "After that, probably you will need to reindex all your index to
> apply the new template."
> Do you have any procedure to do this?
>
>
> Em quarta-feira, 28 de setembro de 2016 18:01:12 UTC-3, jose escreveu:
> >
> > Hi Roberto,
> >
> > Have you applied the custom mapping?
> >
> > http://documentation.wazuh.com/en/latest/ossec_elk_
> > elasticsearch.html#ossec-alerts-template
> >
> > If you have the custom mapping applied, and the template in Logstash, you
> > need to wait until next day, when the next index is created with the new
> > mapping and template.
> >
> > After that, probably you will need to reindex all your index to apply the
> > new template.
> >
> >
> > Regards
> > -----------------------
> > Jose Luis Ruiz
> > Wazuh Inc.
> > jo...@wazuh.com
> >
> > On September 28, 2016 at 3:26:38 PM, roberto....@phoebustecnologia.com.br
> > (roberto....@phoebustecnologia.com.br) wrote:
> >
> > Hi Pedro!
> >
> > I am using the ossec wazuh, I have a question about indexes.
> > I had implemented the logstash without using the file "elastic-ossec-
> > template.json". But I saw it would be good to use it. I am wanting use
> > some indexes and Kibana shows "Analyzed Field", like "AgentName".
> >
> > I put the template in the configuration of logstash and the index has
> > not changed to "not analized".
> >
> >
> > My logstash output :
> >
> > output {
> >
> > #for archives.json log
> > if [type] == "ossecall" {
> > elasticsearch {
> > hosts => "127.0.0.1:9200"
> > index => "ossecall-%{+YYYY.MM.dd}"
> > document_type => "ossecall"
> > template => "/etc/logstash/elastic-ossec-template2.json"
> > template_name => "ossecall"
> > template_overwrite => true
> > }
> > }
> > #for alerts.json log
> > else {
> > elasticsearch {
> > hosts => "127.0.0.1:9200"
> > index => "ossec-%{+YYYY.MM.dd}"
> > document_type => "ossec"
> > template => "/etc/logstash/elastic-ossec-template.json"
> > template_name => "ossec"
> > template_overwrite => true
> > }
> > }
> > }
> >
> > Can you help me?
> >
> >
> >
> > Em quinta-feira, 2 de junho de 2016 08:25:09 UTC-3, Pedro S escreveu:
> > >
> > > Hi Maxim,
> > >
> > > How are you forwarding the alerts/archives to Kibana?
> > >
> > > I think you will need the archives JSON output setting, if you are using
> > > Wazuh <http://wazuh.com/>, edit *ossec.conf* and add the following
> > > setting:
> > >
> > > <global>
> > > > *<logall_json>yes</logall_json>*
> > > > </global>
> > >
> > >
> > >
> > > Once you do it, you will find new archives.json events files at:
> > >
> > > /var/ossec/logs/archives/archives.json
> > >
> > >
> > >
> > > The next step is forward these archives events to Elasticsearch, in
> > > order to do it we need to edit Logstash configuration.
> > >
> > > My personal advice to index archives events is to create a dedicated
> > > index pattern just for them, so you will be able to distinguish between
> > > events and alerts, adding inside "output" section the following
> > > configuration:
> > >
> > > output {
> > > if [type] == "ossec-alerts" {
> > > elasticsearch {
> > > hosts => ["127.0.0.1:9200"]
> > > index => "ossec-%{+YYYY.MM.dd}"
> > > document_type => "ossec"
> > > template => "/etc/logstash/elastic-ossec-template.json"
> > > template_name => "ossec"
> > > template_overwrite => true
> > > }
> > > }
> > > if [type] == "ossec-archives" {
> > > elasticsearch {
> > > hosts => ["127.0.0.1:9200"]
> > > index => "ossec-archives-%{+YYYY.MM.dd}"
> > > document_type => "ossec"
> > > template => "/etc/logstash/elastic-ossec-template.json"
> > > template_name => "ossec"
> > > template_overwrite => true
> > > }
> > > }
> > > }
> > >
> > >
> > > Later in Kibana you will need to create a new index pattern
> > > (Settings->indices) matching for "ossec-archives-*".
> > >
> > > If you need to "reindex" or read the a log file from the beginning using
> > > Logstash, you can use the file input with option *start_position* set
> > > to *beginning* (+ info)
> > > <https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position>
> > >
> > >
> > >
> > > On Monday, May 30, 2016 at 4:53:10 PM UTC+2, Maxim Surdu wrote:
> > > >
> > > > i have this archives files with logs but in kibana i can not see them
> > > > can i reindex this files?
> > > > if i can, please help me step by step
> > > >
> > > > joi, 19 mai 2016, 10:17:51 UTC+3, Maxim Surdu a scris:
> > > > >
> > > > > Hi dear community,
> > > > >
> > > > > i had a problem with logstash, after i resolve it i saw what in kibana
> > > > > are missing logs, how can i resolve the problem and reindexing all my logs
> > > > > to kibana
> > > > > I will be thankful if someone will help me step by step
> > > > >
> > > > >
> > > > > i appreciate your help, and a lot of respect for developers and
> > > > > community!
> > > > >
> > > > --
> >
> > ---
> > You received this message because you are subscribed to the Google Groups
> > "ossec-list" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to ossec-list+...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
> > --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ossec-list+...@googlegroups.com <javascript:>.
> For more options, visit https://groups.google.com/d/optout.
>
> --
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to ossec-list+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups \
"ossec-list" group. To unsubscribe from this group and stop receiving emails from it, \
send an email to ossec-list+unsubscribe@googlegroups.com. For more options, visit \
https://groups.google.com/d/optout.
[Attachment #5 (text/html)]
<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body \
style="word-wrap:break-word"><div id="bloop_customfont" \
style="color:rgb(0,0,0);margin:0px"><font face="Helvetica"><span \
style="font-size:14px">Hi Roberto, nice news :)</span></font></div><div \
id="bloop_customfont" \
style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto"><br></div><div \
id="bloop_customfont" style="color:rgb(0,0,0);margin:0px"><p \
style="margin:0px;line-height:normal"><font face="Helvetica"><span \
style="font-size:14px">Please feel free to send pull request to Wazuh and Ossec with \
your improvements and new rules, the Ossec community will \
appreciate.</span></font></p></div> <div id="bloop_sign_1475247790484785152" \
class="bloop_sign"><div class="" style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word">Regards</div><div class="" \
style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word">-----------------------</div><div \
class="" style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word">Jose Luis Ruiz<br \
class="">Wazuh Inc.<br class=""><a href="mailto:jose@wazuh.com" \
class="">jose@wazuh.com</a></div></div> <br><p class="airmail_on">On September 30, \
2016 at 9:00:32 AM, <a \
href="mailto:roberto.mendonca@phoebustecnologia.com.br">roberto.mendonca@phoebustecnologia.com.br</a> \
(<a href="mailto:roberto.mendonca@phoebustecnologia.com.br">roberto.mendonca@phoebustecnologia.com.br</a>) \
wrote:</p> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>
<title></title>
<div dir="ltr">Hi Jose!<br>
<br>
<span id="result_box" class="" lang="en"><span>The</span> <span \
class="">script</span> <span class="">worked beautifully! rsrs<br> Very \
thanks!<br></span></span><br> <span id="result_box" class="" lang="en"><span>Out of
this</span> <span>topic</span><span>,</span> <span>I'm thinking
of</span> <span>improving</span> <span>the rules for</span>
<span>some</span> <span>Windows</span> <span>security
events</span><span>.</span> <span>I do not know</span> <span>if
there is already</span> <span>a topic</span> <span>or</span>
<span>work on</span> <span>it.</span><br>
<span>For the</span> <span>ossec</span> <span>generate</span>
<span>alerts,</span> <span>for example,</span> <span>the</span>
<span>login types</span><span class="">:<br>
<img src="cid:1DDBB4F5-3148-44BC-8211-2FE243FAFD37" alt=""><br>
<br></span></span><br>
<span id="result_box" class="" lang="en"><span>And
then</span> <span class="">would release</span> <span class="">on
github</span><span>.</span> <span>I</span> <span class="">would
like to contribute</span><span>,</span> <span class="">if
possible</span><span>.<br>
<br></span></span>Em quinta-feira, 29 de setembro de 2016
09:53:05 UTC-3, jose escreveu:
<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px \
#ccc solid;padding-left:1ex"> <div style="word-wrap:break-word">
<div>
<p>Hi Roberto,</p>
<p>About your osseccall you wrote this in the mail</p>
<pre><code>But the file "template =>" \
/etc/logstash/elastic-ossec-<wbr>template2.json "I modified the lines 3 and 8. \
Line 3: from "template", "ossec *" to "template", \
"ossecall *" Line 8: from "ossec": to "ossecall":
</code></pre>
<p>You have an space between ossec, ossecall and the wildcard?, if
you have, you should not. And with the curl procedure:</p>
<pre><code>$ Cd ~ / ossec_tmp / ossec-wazuh / extensions / ElasticSearch / && \
curl -XPUT "http: // localhost: 9200 / _template / ossec /" -d "@ \
elastic-ossec-template.json" </code></pre>
<p>You need to apply the templates for both index.</p>
<p>For your last question, in this mail you have a bash script to
reindex the index. Please use carefully and check with <code>curl
'localhost:9200/_cat/indices?<wbr>v'</code> after every step that
the script is doing well.</p>
<p>This script has 4 steps:</p>
<ol>
<li>We move all index without mapping applied to a backup index, we
do that with the option <code>reindex</code> to apply the new
template.</li>
<li>After the reindex is has finished we can delete the old
index.</li>
<li>Now we can move the backup index to the original name.</li>
<li>When the step 3 has finished we can delete the backup
index.</li>
</ol>
<p>Pleas take a look the lines 72, 73 and 76, 77 in order to change
the index name from <code>ossec-$index_elastic_name</code> and
<code>ossec-$index_elastic_name</code> by
<code>ossecall-$index_elastic_name</code> and
<code>ossecall-$index_elastic_name</code> because probably you need
to run this script for your two index.</p>
<p>This one of a few utils that wazuh will release soon.</p>
<pre><code>#!/bin/bash
# Copyright (C) 2015-2016 Wazuh, Inc.All rights reserved.
# Wazuh.com
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.
# Elasticsearch Reindexing
# Requires:
# Elasticsearch 2.3 or superior
if [ $# -ne 4 ]
then
echo "Usage: ./wazuh_elastic_reindex_index.<wbr>sh date_from date_to \
elasticsearch_ip step" echo -e "\tDate format: YYYY-MM-DD"
echo -e "\tStep: 1|2|3|4"
echo -e "\tExample: ./wazuh_elastic_reindex_index.<wbr>sh 20160826 20160901 \
10.0.0.20 1" echo -e "\tNote: Each step takes its time to perform the \
actions required. Review: tail -f /var/log/elasticsearch/ossec.<wbr>log" exit 0
fi
## Arguments
FROM=$1
TO=$2
ELASTIC_IP=$3
STEP=$4
## Main
startdate=$(date -d $FROM +"%Y%m%d")
enddate=$(date -d $TO +"%Y%m%d")
if [ $startdate -ge $enddate ];
then
echo "The date_from $startdate is bigger than date_to $enddate, please review \
this arguments"; exit 1
fi
startdate=$(date -I -d "$FROM") || exit -1
enddate=$(date -I -d "$TO") || exit -1
echo -e "\n### Start reindexing [STEP $STEP], from $startdate to $enddate are \
you sure? please confirm with YES/NO?" read ADDRANSWER
exist_index () {
request="$ELASTIC_IP:9200/$1"
exist=`curl -s -XHEAD -i $request | head -n 1 | cut -d' ' -f2`
}
reindex () {
request="$ELASTIC_IP:9200/_<wbr>reindex"
request_body='{ "source": { "index": \
"'"$1"'" }, "dest": { "index": \
"'"$2"'" }}' curl_result=`curl -s -XPOST $request -d \
"$request_body"` echo $curl_result
}
delete_index () {
request="$ELASTIC_IP:9200/$1"
curl_result=`curl -s -XDELETE $request`
echo $curl_result
}
if [ $ADDRANSWER == 'YES' ]
then
d="$FROM"
while [ "$d" != "$enddate" ]; do
index_elastic_name=` echo $d | sed 's/-/\./g'`
if [ $STEP == '1' ] || [ $STEP == '2' ]; then
src_index="ossec-$index_<wbr>elastic_name"
dst_index="ossec-$index_<wbr>elastic_name-b"
exist_index $src_index
elif [ $STEP == '3' ] || [ $STEP == '4' ]; then
src_index="ossec-$index_<wbr>elastic_name-b"
dst_index="ossec-$index_<wbr>elastic_name"
exist_index $src_index
else
echo "Bad argument: step: $STEP"
exit 1
fi
if [ $exist != '404' ]; then
if [ $STEP == '1' ]; then
echo "### 1. Reindexing: $src_index -> $dst_index"
reindex $src_index $dst_index
elif [ $STEP == '2' ]; then
echo "### 2. Deleting old index: $src_index"
delete_index $src_index
elif [ $STEP == '3' ]; then
echo "### 3. Reindexing: $src_index"
reindex $src_index $dst_index
elif [ $STEP == '4' ]; then
echo "### 4. Deleting intemediate index: $src_index"
delete_index $src_index
fi
else
echo "### Index $src_index doest not exist. Skipping."
fi
# Update date.
d=$(date -I -d "$d + 1 day")
done
echo -e "\nPlease check 'curl -XGET \
${ELASTIC_IP}:9200/_cat/<wbr>indices' to re-check the indices" echo \
"Reindexing ended [STEP $STEP]." else
echo "This script is finished because you don't confirm with YES"
fi
</code></pre>
<p>i hope this helps.</p>
</div>
<div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div>
<div style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word"> Regards</div>
<div style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word">
-----------------------</div>
<div style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word"> Jose Luis Ruiz<br>
Wazuh Inc.<br>
<a href="javascript:" target="_blank" rel="nofollow" \
onmousedown="this.href='javascript:';return true;" \
onclick="this.href='javascript:';return true;">jo...@wazuh.com</a></div> \
</div> <br>
<p>On September 29, 2016 at 7:25:09 AM, <a href="javascript:" target="_blank" \
rel="nofollow" onmousedown="this.href='javascript:';return true;" \
onclick="this.href='javascript:';return \
true;">roberto....@<wbr>phoebustecnologia.com.br</a> (<a href="javascript:" \
target="_blank" rel="nofollow" onmousedown="this.href='javascript:';return \
true;" onclick="this.href='javascript:';return \
true;">roberto....@<wbr>phoebustecnologia.com.br</a>) wrote:</p>
<blockquote type="cite">
<div>
<div>
<div dir="ltr"><span><span lang="en"><span>Hi Jose,
thanks for reply!<br>
<br>
Indeed,</span> <span>today</span> <span>the</span>
<span>index</span> <span>is in</span> <span>template</span>
<span>format.</span> <span>But</span> <span>only</span>
<span>ossec</span> <span>index, the</span> <span>index</span>
<span>ossecall</span> <span>did not work</span><span>, the
fields</span> <span>still appear</span> <span>as
"</span><span>Analyzed</span>
<span>Field</span><span>"</span><span>.</span><br>
<br>
<span>I did not do</span> <span>the
procedure</span><span>:</span><br>
<span>$</span> <span>Cd ~</span> <span>/</span>
<span>ossec_tmp</span> <span>/</span>
<span>ossec</span><span>-</span><span>wazuh</span> <span>/
extensions /</span> <span>ElasticSearch</span> <span>/</span>
<span>&&</span> <span>curl</span> <span>-XPUT</span>
<span>"http</span><span>: //</span> <span>localhost:</span>
<span>9200</span> <span>/</span> <span>_template</span>
<span>/</span> <span>ossec</span> <span>/</span><span>" -d</span>
<span>"@</span>
<span>elastic-</span><span>ossec</span><span>-</span><span>template.json</span><span>"</span><br>
<br>
<span>Just</span> <span>put</span> <span>the</span>
<span>logstash</span> <span>output</span> <span>that</span> <span>I
said.</span><br>
<br>
<span>But</span> <span>the file "</span><span>template
=</span><span>>"</span>
<span>/etc/logstash/elastic-ossec-<wbr>template<b>2</b>.json</span>
<span>"</span><span>I modified</span> <span>the lines</span>
<span>3 and 8</span><span>.</span><br>
<span>Line 3:</span> <span><b>from</b>
"template</span><span>"</span><span>, "</span><span>ossec</span>
<span>*"</span> <b>to</b> <span>"template"</span><span>,
"</span><span>ossecall</span> <span>*</span><span>"</span><br>
<span>Line</span> <span>8:</span></span> <span \
lang="en"><span><b>from</b></span></span> <span \
lang="en"><span>"</span><span>ossec</span><span>"</span><span>:</span> \
<b><span>to</span></b> \
<span>"</span><span>ossecall</span><span>"</span><span>:</span><br>
<br>
<span>I do not know</span> <span>if it was really</span>
<span>necessary to do this</span><span>.</span></span> <span lang="en"><span><span \
lang="en"><span>I did</span> <span>this because</span> <span>I decided to
create</span> <span>a separate index</span> <span>for</span>
<span>logs</span> <span>archives.json</span> <span>file.</span>
<span>Where</span> <span>ossec</span> <span>are logging</span>
<span>all</span><span>.</span></span><br>
<br>
About "</span></span><span lang="en"><span>After
that, probably you will need to reindex all your index to apply the
new template."<br>
Do you have any procedure to do this?<br>
<br></span></span><br>
Em quarta-feira, 28 de setembro de 2016 18:01:12 UTC-3, jose
escreveu:</span>
<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px \
#ccc solid;padding-left:1ex"> <div style="word-wrap:break-word">
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
Hi Roberto,</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
Have you applied the custom mapping?</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<a href="http://documentation.wazuh.com/en/latest/ossec_elk_elasticsearch.html#ossec-alerts-template" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fdocumentation. \
wazuh.com%2Fen%2Flatest%2Fossec_elk_elasticsearch.html%23ossec-alerts-template\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG5Z198kZoe90LCz9k2bhfSgZY6cw';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fdocumentati \
on.wazuh.com%2Fen%2Flatest%2Fossec_elk_elasticsearch.html%23ossec-alerts-template\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG5Z198kZoe90LCz9k2bhfSgZY6cw';return \
true;"> http://documentation.wazuh.<wbr>com/en/latest/ossec_elk_<wbr>elasticsearch.html#ossec-<wbr>alerts-template</a></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
If you have the custom mapping applied, and the template in
Logstash, you need to wait until next day, when the next index is
created with the new mapping and template.</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
After that, probably you will need to reindex all your index to
apply the new template.</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<br>
<div>
<div style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word"> Regards</div>
<div style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word">
-----------------------</div>
<div style="font-family:'helvetica \
Neue',helvetica;font-size:14px;word-wrap:break-word"> Jose Luis Ruiz<br>
Wazuh Inc.<br>
<a rel="nofollow">jo...@wazuh.com</a></div>
</div>
<br>
<p>On September 28, 2016 at 3:26:38 PM, <a \
rel="nofollow">roberto....@phoebustecnologia.<wbr>com.br</a> (<a \
rel="nofollow">roberto....@<wbr>phoebustecnologia.com.br</a>) wrote:</p>
<blockquote type="cite">
<div>
<div>
<div dir="ltr"><span><span lang="en"><span>Hi
Pedro!</span><br>
<br>
<span>I am</span> <span>using the</span> <span>ossec</span>
<span>wazuh</span><span>, I have a</span> <span>question
about</span> <span>indexes.</span><br>
<span>I had</span> <span>implemented</span> <span>the</span>
<span>logstash</span> <span>without using</span> <span>the file
"</span><span>elastic-</span><span>ossec</span><span>-</span><span>template.json</span><span>"</span><span>.</span>
<span>But</span> <span>I saw</span> <span>it would be good</span>
<span>to use</span> <span>it.</span> <span>I am wanting</span>
<span>use some</span> <span>indexes</span> <span>and</span>
<span>Kibana</span> <span>shows</span>
<span>"</span><span>Analyzed</span>
<span>Field</span><span>"</span><span>,</span> <span>like
"</span><span>AgentName</span><span>"</span><span>.</span><br>
<br>
<span>I put the</span> <span>template</span> <span>in the
configuration of logstash and</span> <span>the index</span>
<span>has not changed</span> <span>to</span> <span>"not</span>
<span>analized</span><span>".</span><br>
<img src="https://groups.google.com/group/ossec-list/attach/b53145d9418d1/autoGeneratedInlineImage1?part=0.1&authuser=0" \
alt=""><br> <br>
My logstash output :<br>
<br>
output {<br>
<br>
#for archives.json log<br>
if [type] == "ossecall" {<br>
elasticsearch {<br>
hosts => "<a href="http://127.0.0.1:9200" rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A92 \
00\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3 \
A9200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;">127.0.0.1:9200</a>"<br>
index => "ossecall-%{+YYYY.MM.dd}"<br>
document_type => "ossecall"<br>
template =>
"/etc/logstash/elastic-ossec-<wbr>template2.json"<br>
template_name => "ossecall"<br>
template_overwrite => true<br>
}<br>
}<br>
#for alerts.json log<br>
else {<br>
elasticsearch {<br>
hosts => "<a href="http://127.0.0.1:9200" rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A92 \
00\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3 \
A9200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;">127.0.0.1:9200</a>"<br>
index => "ossec-%{+YYYY.MM.dd}"<br>
document_type => "ossec"<br>
template =>
"/etc/logstash/elastic-ossec-<wbr>template.json"<br>
template_name => "ossec"<br>
template_overwrite => true<br>
}<br>
}<br>
}<br>
<br>
<span>Can you help me?</span></span><br>
<br>
<br>
<br>
Em quinta-feira, 2 de junho de 2016 08:25:09 UTC-3, Pedro S
escreveu:</span>
<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px \
#ccc solid;padding-left:1ex"> <div dir="ltr">Hi Maxim,
<div><br></div>
<div>How are you forwarding the alerts/archives to Kibana?</div>
<div><br></div>
<div>I think you will need the archives JSON output setting, if you
are using <a href="http://wazuh.com/" rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwazuh.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGUbl5cCaBvcco52JvaMmeUhpieIw';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwazuh.com%2 \
F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGUbl5cCaBvcco52JvaMmeUhpieIw';return \
true;"> Wazuh</a>, edit <i>ossec.conf</i> and add the following
setting:</div>
<div><br></div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<global><br>
<b><logall_json>yes</logall_json></b><br>
</global></blockquote>
<div><br></div>
<div><br></div>
<div>Once you do it, you will find new archives.json events files
at:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
/var/ossec/logs/archives/<wbr>archives.json</blockquote>
</div>
<div><br></div>
<div><br></div>
<div>The next step is forward these archives events to
Elasticsearch, in order to do it we need to edit Logstash
configuration.</div>
<div><br></div>
<div>
<div>My personal advice to index archives events is to create a
dedicated index pattern just for them, so you will be able to
distinguish between events and alerts, adding inside "output"
section the following configuration:</div>
<div><br></div>
<div style="border:1px solid \
rgb(187,187,187);word-wrap:break-word;background-color:rgb(250,250,250)"> \
<div><code><span style="color:#000">output</span> <span \
style="color:#660">{</span><span style="color:#000"><br> </span> <span \
style="color:#008">if</span> <span style="color:#660">[</span><span \
style="color:#000">type</span><span style="color:#660">]</span> <span \
style="color:#660">==</span> <span style="color:#080">"ossec-alerts"</span> \
<span style="color:#660">{</span><span style="color:#000"><br>
elasticsearch</span> <span style="color:#660">{</span><span \
style="color:#000"><br> hosts</span>
<span style="color:#660">=></span> <span style="color:#660">[</span><span \
style="color:#080">"<a href="http://127.0.0.1:9200" rel="nofollow" \
target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F% \
2F127.0.0.1%3A9200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3 \
A9200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;">127.0.0.1:9200</a>"</span><span style="color:#660">]</span><span \
style="color:#000"><br>
index</span>
<span style="color:#660">=></span> <span \
style="color:#080">"ossec-%{+YYYY.MM.dd}"</span><span \
style="color:#000"><br>
document_type</span> <span style="color:#660">=></span>
<span style="color:#080">"ossec"</span><span style="color:#000"><br>
</span><span style="color:#008">template</span> <span \
style="color:#660">=></span> <span \
style="color:#080">"/etc/logstash/elastic-ossec-<wbr>template.json"</span><span \
style="color:#000"><br>
template_name</span> <span style="color:#660">=></span>
<span style="color:#080">"ossec"</span><span style="color:#000"><br>
template_overwrite</span> <span style="color:#660">=></span> <span \
style="color:#008">true</span><span style="color:#000"><br>
</span> <span style="color:#660">}</span><span style="color:#000"><br>
</span> <span style="color:#660">}</span><span style="color:#000"><br>
</span> <span style="color:#008">if</span>
<span style="color:#660">[</span><span style="color:#000">type</span><span \
style="color:#660">]</span> <span style="color:#660">==</span> <span \
style="color:#080">"ossec-archives"</span> <span \
style="color:#660">{</span><span style="color:#000"><br>
elasticsearch</span> <span style="color:#660">{</span><span \
style="color:#000"><br> hosts</span>
<span style="color:#660">=></span> <span style="color:#660">[</span><span \
style="color:#080">"<a href="http://127.0.0.1:9200" rel="nofollow" \
target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F% \
2F127.0.0.1%3A9200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3 \
A9200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH14hW3khm7lTWZyJM1wkUgOtJPTA';return \
true;">127.0.0.1:9200</a>"</span><span style="color:#660">]</span><span \
style="color:#000"><br>
index</span>
<span style="color:#660">=></span> <span \
style="color:#080">"ossec-archives-%{+YYYY.MM.dd}<wbr>"</span><span \
style="color:#000"><br>
document_type</span> <span style="color:#660">=></span>
<span style="color:#080">"ossec"</span><span style="color:#000"><br>
</span><span style="color:#008">template</span> <span \
style="color:#660">=></span> <span \
style="color:#080">"/etc/logstash/elastic-ossec-<wbr>template.json"</span><span \
style="color:#000"><br>
template_name</span> <span style="color:#660">=></span>
<span style="color:#080">"ossec"</span><span style="color:#000"><br>
template_overwrite</span> <span style="color:#660">=></span> <span \
style="color:#008">true</span><span style="color:#000"><br>
</span> <span style="color:#660">}</span><span style="color:#000"><br>
</span> <span style="color:#660">}</span><span \
style="color:#000"><br></span><span style="color:#660">}</span></code></div> </div>
<div><br></div>
</div>
<div><br></div>
<div>Later in Kibana you will need to create a new index pattern
(Settings->indices) matching for "ossec-archives-*".<br></div>
<div><br></div>
<div>If you need to "reindex" or read the a log file from the
beginning using Logstash, you can use the file input with option
<i>start_position</i> set to <i>beginning</i> <a \
href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position" \
rel="nofollow" target="_blank" \
onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.elastic. \
co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-inputs-file.html%23plugins-inputs-file- \
start_position\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEjE1gfMJ_xpLo9MfHAbHd9mmZowA';return \
true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.elast \
ic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-inputs-file.html%23plugins-inputs-fi \
le-start_position\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEjE1gfMJ_xpLo9MfHAbHd9mmZowA';return \
true;"> (+ info)</a><br>
<br>
<br></div>
<br>
On Monday, May 30, 2016 at 4:53:10 PM UTC+2, Maxim Surdu wrote:
<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px \
#ccc solid;padding-left:1ex"> <div dir="ltr">i have this archives files with logs but \
in kibana i can not see them can i reindex this files?<br>
if i can, please help me step by step<br>
<br>
joi, 19 mai 2016, 10:17:51 UTC+3, Maxim Surdu a scris:
<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px \
#ccc solid;padding-left:1ex"> <div dir="ltr">Hi dear community,<br>
<br>
<div>i had a problem with logstash, after i resolve it i saw what
in kibana are missing logs, how can i resolve the problem and
reindexing all my logs to kibana<br>
I will be thankful if someone will help me step by step<br></div>
<div><br></div>
<div><br>
i appreciate your help, and a lot of respect for developers and
community!</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
--<br>
<br>
---<br>
You received this message because you are subscribed to the Google
Groups "ossec-list" group.<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a rel="nofollow">ossec-list+...@googlegroups.<wbr>com</a>.<br>
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" \
target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return \
true;" onclick="this.href='https://groups.google.com/d/optout';return \
true;">https://groups.google.com/d/<wbr>optout</a>.<br> </div>
</div>
</blockquote>
</div>
</blockquote>
</div>
--<br>
<br>
---<br>
You received this message because you are subscribed to the Google
Groups "ossec-list" group.<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a href="javascript:" target="_blank" rel="nofollow" \
onmousedown="this.href='javascript:';return true;" \
onclick="this.href='javascript:';return \
true;">ossec-list+...@<wbr>googlegroups.com</a>.<br>
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" \
rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return \
true;" onclick="this.href='https://groups.google.com/d/optout';return \
true;">https://groups.google.com/d/<wbr>optout</a>.<br> </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
--<br>
<br>
---<br>
You received this message because you are subscribed to the Google
Groups "ossec-list" group.<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a href="mailto:ossec-list+unsubscribe@googlegroups.com">ossec-list+unsubscribe@googlegroups.com</a>.<br>
For more options, visit <a \
href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br>
</div></div></span></blockquote></body></html>
<p></p>
-- <br />
<br />
--- <br />
You received this message because you are subscribed to the Google Groups \
"ossec-list" group.<br /> To unsubscribe from this group and stop receiving \
emails from it, send an email to <a \
href="mailto:ossec-list+unsubscribe@googlegroups.com">ossec-list+unsubscribe@googlegroups.com</a>.<br \
/> For more options, visit <a \
href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br \
/>
--001a1140eb440be527053dbbcaf8--
["autoGeneratedInlineImage1" (application/octet-stream)]
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic