[prev in list] [next in list] [prev in thread] [next in thread]
List: security-onion
Subject: Re: [security-onion] Logstash - database causing crash
From: Wes Lambert <wlambertts () gmail ! com>
Date: 2018-09-25 14:56:27
Message-ID: CAHjBB6F1x2zM2Y7+xzBjMBKPPJUCoJXzMFH5PitM9nbqorcBzw () mail ! gmail ! com
[Download RAW message or body]
You need to use a volume to mount the path and then refer to the path that
is relative to the containers view:
So, using LOGSTASH_OPTIONS in /etc/nsm/securityonion.conf, if you do
something like:
--volume /etc/logstash/mynewgoeipdatabase.mmdb:/mynewgeoipdatabase.mmdb:ro
Then you would refer to it in the config like:
geoip {
source => "source_ip" database => "/mynewgeoipdatabase.mmdb" }
If you need further assistance, it may behoove you to post on the Elastic
forum(s) as well.
https://discuss.elastic.co/
Thanks,
Wes
On Mon, Sep 24, 2018 at 5:24 PM Petr Řeřicha <petr.rericha1@gmail.com>
wrote:
> Dne pondělí 24. září 2018 7:46:56 UTC-5 Wes napsal(a):
> > You should be able to override the database path by explicitly declaring
> it in the config:
> >
> >
> >
> https://discuss.elastic.co/t/how-to-logstash-geoip-database-file-update/91359
> >
> >
> >
> > Thanks,
> > Wes
> >
> >
> > On Fri, Sep 21, 2018 at 11:45 AM Petr Řeřicha <petr.r...@gmail.com>
> wrote:
> > Dne čtvrtek 20. září 2018 11:23:15 UTC-5 Wes napsal(a):
> >
> > > Hi Petr,
> >
> > >
> >
> > >
> >
> > > Is there any reason, you don't want to use the existing source_ip and
> destination_ip fields (which are already using geoip), or the default
> GeoLiteCity2 DB?
> >
> > >
> >
> > >
> >
> > >
> https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html
> >
> > >
> >
> > >
> >
> > > Thanks,
> >
> > > Wes
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > On Tue, Sep 18, 2018 at 2:08 PM Petr Řeřicha <petr.r...@gmail.com>
> wrote:
> >
> > > Hello everybody,
> >
> > >
> >
> > > I'd like to ask how can I use database for geoip filter. I've created
> new config file for filtering logs, which worked pretty well I guess. Then
> I tried to redirect to another database GeoLiteCity.dat in /etc/logstash/
> writing this in my filter database => "/etc/logstash/GeoLiteCity.dat and it
> showed me error in Logstash's file telling me that path is wrong or the
> file could not be read. I even tried the other database format .mmdb and
> it's not working for me either. Logstash is running on Docker image and
> /usr/share/logstash folders are missing. But it still works, I suppose
> Docker uses some symlinks or something (maybe this files
> /usr/share/GeoIP/*). My actual question would be, where should I put my
> .dat file to make it read that geoip database and is it possible to edit
> MaxMind's database somehow (.dat). Thanks for your help :)
> >
> > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > >
> >
> > > Follow Security Onion on Twitter!
> >
> > >
> >
> > > https://twitter.com/securityonion
> >
> > >
> >
> > > ---
> >
> > >
> >
> > > You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> >
> > >
> >
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com.
> >
> > >
> >
> > > To post to this group, send email to securit...@googlegroups.com.
> >
> > >
> >
> > > Visit this group at https://groups.google.com/group/security-onion.
> >
> > >
> >
> > > For more options, visit https://groups.google.com/d/optout.
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > >
> >
> > >
> >
> > > https://twitter.com/therealwlambert
> >
> > >
> >
> > > https://securityonion.net/
> >
> >
> >
> > Hello Wess,
> >
> > I'm using those fields, but I'd like to edit geoip database because it
> is very inaccurate. I've already found folders and even database files in
> Docker image and now I'm trying to figure out how to use another database
> which is already edited by me. Do you know where to find defined path to
> the database which is being used? Even if I redirect path in my config file
> to Docker's image database which is already used, it won't work and
> Logstash crashes while starting.
> >
> > Thanks
> >
> >
> >
> > --
> >
> > Follow Security Onion on Twitter!
> >
> > https://twitter.com/securityonion
> >
> > ---
> >
> > You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> >
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com.
> >
> > To post to this group, send email to securit...@googlegroups.com.
> >
> > Visit this group at https://groups.google.com/group/security-onion.
> >
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> >
> >
> > --
> >
> >
> > https://twitter.com/therealwlambert
> >
> > https://securityonion.net/
>
> Yes, I should be able to do that, but it seems to cause the problem. If I
> use .dat or even newer one .mmdb it says that path is wrong or file cannot
> be opened or something like that. Logstash is one of the containers in
> docker so it might cause problem too.
>
> Thanks for your patience,
> Petr
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
>
--
https://twitter.com/therealwlambert
https://securityonion.net/
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups \
"security-onion" group. To unsubscribe from this group and stop receiving emails from \
it, send an email to security-onion+unsubscribe@googlegroups.com. To post to this \
group, send email to security-onion@googlegroups.com. Visit this group at \
https://groups.google.com/group/security-onion. For more options, visit \
https://groups.google.com/d/optout.
[Attachment #3 (text/html)]
<div dir="ltr"><div dir="ltr">You need to use a volume to mount the path and then \
refer to the path that is relative to the containers view:<div><br></div><div>So, \
using LOGSTASH_OPTIONS in /etc/nsm/securityonion.conf, if you do something \
like:</div><div><br></div><div><span \
style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation \
Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre">--volume \
/etc/logstash/mynewgoeipdatabase.mmdb:/mynewgeoipdatabase.mmdb:ro</span></div><div><br></div><div>Then \
you would refer to it in the config like:</div><div><br></div><div> <span \
style="background-color:rgb(240,240,240);color:rgb(51,51,51);font-family:Consolas,Menlo,Monaco,"Lucida \
Console","Liberation Mono","DejaVu Sans \
Mono","Bitstream Vera Sans Mono","Courier \
New",monospace;font-size:14px;white-space:pre">geoip {</span><br></div><span \
style="color:rgb(51,51,51);font-family:Consolas,Menlo,Monaco,"Lucida \
Console","Liberation Mono","DejaVu Sans \
Mono","Bitstream Vera Sans Mono","Courier \
New",monospace;font-size:14px;white-space:pre;background-color:rgb(240,240,240)"> \
source => </span><span class="gmail-hljs-string" \
style="color:rgb(221,17,68);font-family:Consolas,Menlo,Monaco,"Lucida \
Console","Liberation Mono","DejaVu Sans \
Mono","Bitstream Vera Sans Mono","Courier \
New",monospace;font-size:14px;white-space:pre">"source_ip"</span><span \
style="color:rgb(51,51,51);font-family:Consolas,Menlo,Monaco,"Lucida \
Console","Liberation Mono","DejaVu Sans \
Mono","Bitstream Vera Sans Mono","Courier \
New",monospace;font-size:14px;white-space:pre;background-color:rgb(240,240,240)">
database => </span><span class="gmail-hljs-string" \
style="color:rgb(221,17,68);font-family:Consolas,Menlo,Monaco,"Lucida \
Console","Liberation Mono","DejaVu Sans \
Mono","Bitstream Vera Sans Mono","Courier \
New",monospace;font-size:14px;white-space:pre">"/mynewgeoipdatabase.mmdb"</span><span \
style="color:rgb(51,51,51);font-family:Consolas,Menlo,Monaco,"Lucida \
Console","Liberation Mono","DejaVu Sans \
Mono","Bitstream Vera Sans Mono","Courier \
New",monospace;font-size:14px;white-space:pre;background-color:rgb(240,240,240)">
}</span><div><div><br></div><div>If you need further assistance, it may behoove you \
to post on the Elastic forum(s) as well.</div><div><br></div><div><a \
href="https://discuss.elastic.co/">https://discuss.elastic.co/</a><br></div><div><br></div><div>Thanks,</div><div>Wes</div><div><br></div><br \
class="gmail-Apple-interchange-newline"></div></div></div><br><div \
class="gmail_quote"><div dir="ltr">On Mon, Sep 24, 2018 at 5:24 PM Petr Řeřicha \
<<a href="mailto:petr.rericha1@gmail.com">petr.rericha1@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex">Dne pondělí 24. září 2018 \
7:46:56 UTC-5 Wes napsal(a):<br> > You should be able to override the database \
path by explicitly declaring it in the config:<br> > <br>
> <br>
> <a href="https://discuss.elastic.co/t/how-to-logstash-geoip-database-file-update/91359" \
rel="noreferrer" target="_blank">https://discuss.elastic.co/t/how-to-logstash-geoip-database-file-update/91359</a><br>
> <br>
> <br>
> <br>
> Thanks,<br>
> Wes<br>
> <br>
> <br>
> On Fri, Sep 21, 2018 at 11:45 AM Petr Řeřicha <<a \
href="mailto:petr.r...@gmail.com" target="_blank">petr.r...@gmail.com</a>> \
wrote:<br> > Dne čtvrtek 20. září 2018 11:23:15 UTC-5 Wes napsal(a):<br>
> <br>
> > Hi Petr,<br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > Is there any reason, you don't want to use the existing source_ip and \
destination_ip fields (which are already using geoip), or the default GeoLiteCity2 \
DB?<br> > <br>
> > <br>
> <br>
> > <br>
> <br>
> > <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html" \
rel="noreferrer" target="_blank">https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html</a><br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > Thanks,<br>
> <br>
> > Wes<br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > On Tue, Sep 18, 2018 at 2:08 PM Petr Řeřicha <<a \
href="mailto:petr.r...@gmail.com" target="_blank">petr.r...@gmail.com</a>> \
wrote:<br> > <br>
> > Hello everybody,<br>
> <br>
> > <br>
> <br>
> > I'd like to ask how can I use database for geoip filter. I've \
created new config file for filtering logs, which worked pretty well I guess. Then I \
tried to redirect to another database GeoLiteCity.dat in /etc/logstash/ writing this \
in my filter database => "/etc/logstash/GeoLiteCity.dat and it showed me \
error in Logstash's file telling me that path is wrong or the file could not be \
read. I even tried the other database format .mmdb and it's not working for me \
either. Logstash is running on Docker image and /usr/share/logstash folders are \
missing. But it still works, I suppose Docker uses some symlinks or something (maybe \
this files /usr/share/GeoIP/*). My actual question would be, where should I put my \
.dat file to make it read that geoip database and is it possible to edit \
MaxMind's database somehow (.dat). Thanks for your help :)<br> > <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > -- <br>
> <br>
> > <br>
> <br>
> > Follow Security Onion on Twitter!<br>
> <br>
> > <br>
> <br>
> > <a href="https://twitter.com/securityonion" rel="noreferrer" \
target="_blank">https://twitter.com/securityonion</a><br> > <br>
> > <br>
> <br>
> > --- <br>
> <br>
> > <br>
> <br>
> > You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br> > <br>
> > <br>
> <br>
> > To unsubscribe from this group and stop receiving emails from it, send an \
email to <a href="mailto:security-onio...@googlegroups.com" \
target="_blank">security-onio...@googlegroups.com</a>.<br> > <br>
> > <br>
> <br>
> > To post to this group, send email to <a \
href="mailto:securit...@googlegroups.com" \
target="_blank">securit...@googlegroups.com</a>.<br> > <br>
> > <br>
> <br>
> > Visit this group at <a \
href="https://groups.google.com/group/security-onion" rel="noreferrer" \
target="_blank">https://groups.google.com/group/security-onion</a>.<br> > <br>
> > <br>
> <br>
> > For more options, visit <a href="https://groups.google.com/d/optout" \
rel="noreferrer" target="_blank">https://groups.google.com/d/optout</a>.<br> > \
<br> > > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > -- <br>
> <br>
> > <br>
> <br>
> > <br>
> <br>
> > <a href="https://twitter.com/therealwlambert" rel="noreferrer" \
target="_blank">https://twitter.com/therealwlambert</a><br> > <br>
> > <br>
> <br>
> > <a href="https://securityonion.net/" rel="noreferrer" \
target="_blank">https://securityonion.net/</a><br> > <br>
> <br>
> <br>
> Hello Wess,<br>
> <br>
> I'm using those fields, but I'd like to edit geoip database because it \
is very inaccurate. I've already found folders and even database files in Docker \
image and now I'm trying to figure out how to use another database which is \
already edited by me. Do you know where to find defined path to the database which is \
being used? Even if I redirect path in my config file to Docker's image database \
which is already used, it won't work and Logstash crashes while starting. <br> \
> <br> > Thanks <br>
> <br>
> <br>
> <br>
> -- <br>
> <br>
> Follow Security Onion on Twitter!<br>
> <br>
> <a href="https://twitter.com/securityonion" rel="noreferrer" \
target="_blank">https://twitter.com/securityonion</a><br> > <br>
> --- <br>
> <br>
> You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br> > <br>
> To unsubscribe from this group and stop receiving emails from it, send an email \
to <a href="mailto:security-onio...@googlegroups.com" \
target="_blank">security-onio...@googlegroups.com</a>.<br> > <br>
> To post to this group, send email to <a \
href="mailto:securit...@googlegroups.com" \
target="_blank">securit...@googlegroups.com</a>.<br> > <br>
> Visit this group at <a href="https://groups.google.com/group/security-onion" \
rel="noreferrer" target="_blank">https://groups.google.com/group/security-onion</a>.<br>
> <br>
> For more options, visit <a href="https://groups.google.com/d/optout" \
rel="noreferrer" target="_blank">https://groups.google.com/d/optout</a>.<br> > \
<br> > <br>
> <br>
> <br>
> -- <br>
> <br>
> <br>
> <a href="https://twitter.com/therealwlambert" rel="noreferrer" \
target="_blank">https://twitter.com/therealwlambert</a><br> > <br>
> <a href="https://securityonion.net/" rel="noreferrer" \
target="_blank">https://securityonion.net/</a><br> <br>
Yes, I should be able to do that, but it seems to cause the problem. If I use .dat or \
even newer one .mmdb it says that path is wrong or file cannot be opened or something \
like that. Logstash is one of the containers in docker so it might cause problem too. \
<br> <br>
Thanks for your patience, <br>
Petr<br>
<br>
-- <br>
Follow Security Onion on Twitter!<br>
<a href="https://twitter.com/securityonion" rel="noreferrer" \
target="_blank">https://twitter.com/securityonion</a><br>
--- <br>
You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br> To unsubscribe from this group and stop \
receiving emails from it, send an email to <a \
href="mailto:security-onion%2Bunsubscribe@googlegroups.com" \
target="_blank">security-onion+unsubscribe@googlegroups.com</a>.<br> To post to this \
group, send email to <a href="mailto:security-onion@googlegroups.com" \
target="_blank">security-onion@googlegroups.com</a>.<br> Visit this group at <a \
href="https://groups.google.com/group/security-onion" rel="noreferrer" \
target="_blank">https://groups.google.com/group/security-onion</a>.<br> For more \
options, visit <a href="https://groups.google.com/d/optout" rel="noreferrer" \
target="_blank">https://groups.google.com/d/optout</a>.<br> </blockquote></div><br \
clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr"><span style="font-size:12.8px"><a \
href="https://twitter.com/therealwlambert" \
target="_blank">https://twitter.com/therealwlambert</a></span><br><div><span \
style="font-size:12.8px"><a href="https://securityonion.net/" \
target="_blank">https://securityonion.net/</a></span><br></div></div></div>
<p></p>
-- <br />
Follow Security Onion on Twitter!<br />
<a href="https://twitter.com/securityonion">https://twitter.com/securityonion</a><br \
/>
--- <br />
You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br /> To unsubscribe from this group and stop \
receiving emails from it, send an email to <a \
href="mailto:security-onion+unsubscribe@googlegroups.com">security-onion+unsubscribe@googlegroups.com</a>.<br \
/> To post to this group, send email to <a \
href="mailto:security-onion@googlegroups.com">security-onion@googlegroups.com</a>.<br \
/> Visit this group at <a \
href="https://groups.google.com/group/security-onion">https://groups.google.com/group/security-onion</a>.<br \
/> For more options, visit <a \
href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br \
/>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic