[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] [FORGED] Elasticscearh-http dest wish list
From: Russell Fulton <r.fulton () auckland ! ac ! nz>
Date: 2019-09-02 20:37:38
Message-ID: 901D937C-EAD7-4B82-AF25-A5F90E05F029 () auckland ! ac ! nz
[Download RAW message or body]
Thanks to both of you :). Fabien is right I was wondering if there was something I \
could do on the syslog-ng side to control the index creation.
> On 3/09/2019, at 3:06 AM, Attila Szakacs (aszakacs) \
> <Attila.Szakacs@oneidentity.com> wrote:
> Thanks Fabien, I think I understand now! π
>
> Answering to Russel:
>
> As far as I know it is not possible to change the mapping type of an already \
> created field in an already created index: \
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#update-mapping
> When started, syslog-ng does not create the index in ES, it relies on ES to create \
> it itself with the default mapping types. If you want to have an index with custom \
> mappings, you will have to create it yourself, before sending logs to it from \
> syslog-ng.
So if I create an index in ES with the appropriate mapping then it will work. I do \
this for another thing I use with ES but that does not have daily indexes just a \
single one. I will have a play and report back with the results β hopefully with \
some useful code ;). I can live with thisβ¦
We have some ES experts in house so I will consult.
> I can come up with a possible enhancement:
> We could give the user an option, to set multiple field mapping types when \
> configuring the elasticsearch-http() destination, and if it is set, syslog-ng will \
> try to create the index with the given mapping types before sending the logs. \
> Although, it does not fit really well with the current implementation of \
> elasticsearch-http(), it might be possible, that we can make it work.
> What do you think about this idea? Is this what you are looking for? π
this is what I was hoping for ;). Even better if destination code know how the \
fields were parsed then set them by default. As a software developer for the last \
40 odd years I realise that that information probably is not available to the \
destination interface and that it would be a non trivial to retrofit.
Having IP addresses indexed as such is vital for what I am doing as it allow searches \
by CIDR blocks etc. Same goes for dates and timestamps.
>
> Best regards,
> Attila
> From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli \
> <wernli@in2p3.fr>
> Sent: Monday, September 2, 2019 10:26 AM
> To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu>
> Subject: Re: [syslog-ng] Elasticscearh-http dest wish list
>
> CAUTION: This email originated from outside of the organization. Do not follow \
> guidance, click links, or open attachments unless you recognize the sender and know \
> the content is safe.
>
> Hi,
>
> On Mon, Sep 02, 2019 at 08:08:03AM +0000, Attila Szakacs (aszakacs) wrote:
> > Please correct me, if I misunderstood something.
>
> I think you misunderstood :)
> Russel was talking about the ES side of things : ES templates.
> The latter define the data types of fields in Elasticsearch.
>
> See
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.elastic.co%2Fg \
> uide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Findices-templates.html&data=02 \
> %7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369 \
> b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=wPPnf6uO4gKDYZT1bmZNwAa1dTeBTcOuvg5UxLLUKEE%3D&reserved=0
>
> ______________________________________________________________________________
> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists \
> .balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40one \
> identity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7 \
> C0%7C0%7C637030095857078847&sdata=nhXDR7qKdda4%2Btxq5PG8%2B3TPWnPLxB5z7v4R%2B%2FuJyKc%3D&reserved=0
>
> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww. \
> balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAt \
> tila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c \
> 989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=FL8H7deOLn5iDDxURGTz4QYYg2CYcOT5g3DX2NHZftw%3D&reserved=0
>
> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.co \
> m%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa31 \
> 8fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=uFc4qtRKfVG2FLVWWzcz4ndyibDPcfl8lYT3sS8U9zA%3D&reserved=0
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic