[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] Disk-buffer in kafka-destinations
From: Ronald Fenner <rfenner () gamecircus ! com>
Date: 2020-01-10 18:40:55
Message-ID: 3B99DCBF-69F8-414B-A596-F4A8447A9F1D () gamecircus ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Yep that was it. Thanks.
Ronald Fenner
Network Architect
Game Circus LLC.
rfenner@gamecircus.com
> On Jan 10, 2020, at 12:09 PM, Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com> \
> wrote:
> Hello,
>
> You have a typo `mem-buff-size` -> `mem-buf-size`.
> Let us know if you have further trouble.
>
> --
> Kokan
> On Fri, Jan 10, 2020 at 10:38:50AM -0600, Ronald Fenner wrote:
> > CAUTION: This email originated from outside of the organization. Do not follow \
> > guidance, click links, or open attachments unless you recognize the sender and \
> > know the content is safe.
> >
> > I'm using
> > syslog-ng 3 (3.23.1)
> > Config version: 3.23
> > Installer-Version: 3.23.1
> > Revision:
> > Compile-Date: Oct 21 2019 21:48:06
> > Module-Directory: /opt/syslog-ng/lib/syslog-ng
> > Module-Path: /opt/syslog-ng/lib/syslog-ng
> > Include-Path: /opt/syslog-ng/share/syslog-ng/include
> > Error opening plugin module; module='mod-java', error='libjvm.so: cannot open \
> > shared object file: No such file or directory' Error opening plugin module; \
> > module='kafka', error='librdkafka.so.1: cannot open shared object file: No such \
> > file or directory'
> > Available-Modules: \
> > add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef, \
> > confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite \
> > ,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudof \
> > ile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
> >
> > Enable-Debug: off
> > Enable-GProf: off
> > Enable-Memtrace: off
> > Enable-IPv6: on
> > Enable-Spoof-Source: off
> > Enable-TCP-Wrapper: off
> > Enable-Linux-Caps: off
> > Enable-Systemd: off
> >
> > I'm using the java version of Kafka as the C version was recently released and I \
> > haven't upped my version of syslog-ng yet.
> > My kafka destination looks like:
> >
> > destination d_kafka_environment {
> > kafka (
> > client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/")
> > kafka-bootstrap-servers("kafka.service.consul:9092")
> > topic("${topic}.${environment}")
> > template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
> > );
> > };
> >
> > I had tried placing the disk-buffer inside the kafka block and inside the \
> > destination block without success.
> > I was using this block for the disk-buffer
> > # disk-buffer(
> > # mem-buff-size(10000000)
> > # disk-buf-size(10000000000)
> > # reliable(yes)
> > # dir("/opt/syslog-ng/disk-buffers/kafka_environment")
> > # )
> >
> > Ronald Fenner
> > Network Architect
> > Game Circus LLC.
> >
> > rfenner@gamecircus.com
> >
> > > On Jan 10, 2020, at 1:08 AM, Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com> \
> > > wrote:
> > > Hello,
> > >
> > > Every destination has a queue by default memory, and it is possible to replace \
> > > with disk based buffer. If that does not work that is probably a bug. Could you \
> > > please write the version of syslog-ng you are using ? Do you use the java or \
> > > the C based kafka destination ?
> > > Please include the config and error.
> > >
> > > --
> > > Kokan
> > >
> > > On Thu, Jan 09, 2020 at 03:07:44PM -0600, Ronald Fenner wrote:
> > > > CAUTION: This email originated from outside of the organization. Do not \
> > > > follow guidance, click links, or open attachments unless you recognize the \
> > > > sender and know the content is safe.
> > > > I can't see to me get syslog-ng to start if I add the disk-buffer to the \
> > > > destination. From the docs page it looks like it should be added to the kafka \
> > > > section but it doesn't work and doesn't work if I put it with in the \
> > > > destination block.
> > > > Is this tied to a specific destination type?
> > > >
> > > > We are trying to make out messages going to kafka more reliable when the \
> > > > cluster is unavailable.
> > > > Ronald Fenner
> > > > Network Architect
> > > > Game Circus LLC.
> > > >
> > > > rfenner@gamecircus.com<mailto:rfenner@gamecircus.com>
> > > >
> > >
> > > > ______________________________________________________________________________
> > > > Member info: \
> > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabi \
> > > > t.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneiden \
> > > > tity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603 \
> > > > %7C0%7C0%7C637142711408966399&sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&reserved=0 \
> > > > <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balab \
> > > > it.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneide \
> > > > ntity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec60660 \
> > > > 3%7C0%7C0%7C637142711408966399&sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&reserved=0>
> > > >
> > > > Documentation: \
> > > > https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.c \
> > > > om%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter \
> > > > .kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c \
> > > > 989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&reserved=0 \
> > > > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
> > > > com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpete \
> > > > r.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439 \
> > > > c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&reserved=0>
> > > >
> > > > FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bala \
> > > > bit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.co \
> > > > m%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C \
> > > > 0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0 \
> > > > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
> > > > com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C \
> > > > c2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C \
> > > > 637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0>
> > > >
> > > ______________________________________________________________________________
> > > Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl \
> > > ists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai% \
> > > 40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867e \
> > > c606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0 \
> > > <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit \
> > > .hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentit \
> > > y.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0% \
> > > 7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0>
> > >
> > > Documentation: \
> > > https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com \
> > > %2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kok \
> > > ai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c18 \
> > > 67ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0 \
> > > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.co \
> > > m%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.ko \
> > > kai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1 \
> > > 867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0>
> > >
> > > FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabi \
> > > t.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C \
> > > c2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C63 \
> > > 7142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0 \
> > > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.co \
> > > m%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc \
> > > 9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142 \
> > > 711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0>
> > >
> >
> > ______________________________________________________________________________
> > Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis \
> > ts.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40on \
> > eidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec60660 \
> > 3%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0 \
> > <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.h \
> > u%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.co \
> > m%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C \
> > 637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0>
> >
> > Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww \
> > w.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01% \
> > 7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e4 \
> > 39c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0 \
> > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com% \
> > 2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai% \
> > 40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec6 \
> > 06603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0>
> >
> > FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
> > com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc \
> > 9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C63714271 \
> > 1408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0 \
> > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com% \
> > 2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de4 \
> > 6a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408 \
> > 976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0> \
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng \
> <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng \
> <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq \
> <http://www.balabit.com/wiki/syslog-ng-faq>
[Attachment #5 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; line-break: after-white-space;" class="">Yep that was it. Thanks.<div \
class=""><br class=""><div class=""> <div style="color: rgb(0, 0, 0); font-family: \
Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: \
normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: \
normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: \
-webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; \
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div \
style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; \
font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: \
normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; \
line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; \
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); \
font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; \
font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: \
normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; \
text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: \
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: \
break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" \
class=""><div class=""><div class="">Ronald Fenner</div><div class="">Network \
Architect</div><div class="">Game Circus LLC.</div></div><div class=""><br \
class=""></div><div class=""><a href="mailto:rfenner@gamecircus.com" \
class="">rfenner@gamecircus.com</a></div></div></div></div> </div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 10, 2020, at \
12:09 PM, Peter Kokai (pkokai) <<a href="mailto:Peter.Kokai@oneidentity.com" \
class="">Peter.Kokai@oneidentity.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, \
0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: \
normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline \
!important;" class="">Hello,</span><br style="caret-color: rgb(0, 0, 0); font-family: \
Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; \
font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
13px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" class="">You have a \
typo `mem-buff-size` -> `mem-buf-size`.</span><br style="caret-color: rgb(0, 0, \
0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: \
normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">Let us know if you have further \
trouble.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; \
font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: \
Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; \
font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">--</span><br style="caret-color: \
rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">Kokan</span><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
13px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" class="">On Fri, Jan \
10, 2020 at 10:38:50AM -0600, Ronald Fenner wrote:</span><br style="caret-color: \
rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote \
type="cite" style="font-family: Helvetica; font-size: 13px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: \
auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">CAUTION: This email \
originated from outside of the organization. Do not follow guidance, click links, or \
open attachments unless you recognize the sender and know the content is safe.<br \
class=""><br class=""><br class="">I'm using<br class="">syslog-ng 3 (3.23.1)<br \
class="">Config version: 3.23<br class="">Installer-Version: 3.23.1<br \
class="">Revision:<br class="">Compile-Date: Oct 21 2019 21:48:06<br \
class="">Module-Directory: /opt/syslog-ng/lib/syslog-ng<br class="">Module-Path: \
/opt/syslog-ng/lib/syslog-ng<br class="">Include-Path: \
/opt/syslog-ng/share/syslog-ng/include<br class="">Error opening plugin module; \
module='mod-java', error='libjvm.so: cannot open shared object file: No such file or \
directory'<br class="">Error opening plugin module; module='kafka', \
error='librdkafka.so.1: cannot open shared object file: No such file or directory'<br \
class="">Available-Modules: \
add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,conf \
gen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite,hook-co \
mmands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml<br \
class="">Enable-Debug: off<br class="">Enable-GProf: off<br class="">Enable-Memtrace: \
off<br class="">Enable-IPv6: on<br class="">Enable-Spoof-Source: off<br \
class="">Enable-TCP-Wrapper: off<br class="">Enable-Linux-Caps: off<br \
class="">Enable-Systemd: off<br class=""><br class="">I'm using the java version of \
Kafka as the C version was recently released and I haven't upped my version of \
syslog-ng yet.<br class=""><br class="">My kafka destination looks like:<br \
class=""><br class="">destination d_kafka_environment {<br \
class=""> kafka (<br \
class=""> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/")<br \
class=""> kafka-bootstrap-servers("kafka.service.consul:9092")<br \
class=""> topic("${topic}.${environment}")<br \
class=""> template("$(format-json --scope \
nv_pairs --exclude MESSAGE)\n")<br class=""> );<br class="">};<br \
class=""><br class="">I had tried placing the disk-buffer inside the kafka block and \
inside the destination block without success.<br class=""><br class="">I was using \
this block for the disk-buffer<br class=""># disk-buffer(<br \
class=""># mem-buff-size(10000000)<br \
class=""># disk-buf-size(10000000000)<br \
class=""># reliable(yes)<br class=""># \
dir("/opt/syslog-ng/disk-buffers/kafka_environment")<br \
class=""># )<br class=""><br class="">Ronald Fenner<br \
class="">Network Architect<br class="">Game Circus LLC.<br class=""><br class=""><a \
href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a><br \
class=""><br class=""><blockquote type="cite" class="">On Jan 10, 2020, at 1:08 AM, \
Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com> wrote:<br class=""><br \
class="">Hello,<br class=""><br class="">Every destination has a queue by default \
memory, and it is possible to replace with disk based buffer. If that does not work \
that is probably a bug.<br class="">Could you please write the version of syslog-ng \
you are using ?<br class="">Do you use the java or the C based kafka destination ?<br \
class=""><br class="">Please include the config and error.<br class=""><br \
class="">--<br class="">Kokan<br class=""><br class="">On Thu, Jan 09, 2020 at \
03:07:44PM -0600, Ronald Fenner wrote:<br class=""><blockquote type="cite" \
class="">CAUTION: This email originated from outside of the organization. Do not \
follow guidance, click links, or open attachments unless you recognize the sender and \
know the content is safe.<br class=""><br class="">I can't see to me get syslog-ng to \
start if I add the disk-buffer to the destination.<br class="">From the docs page it \
looks like it should be added to the kafka section but it doesn't work and doesn't \
work if I put it with in the destination block.<br class=""><br class="">Is this tied \
to a specific destination type?<br class=""><br class="">We are trying to make out \
messages going to kafka more reliable when the cluster is unavailable.<br \
class=""><br class="">Ronald Fenner<br class="">Network Architect<br class="">Game \
Circus LLC.<br class=""><br \
class="">rfenner@gamecircus.com<mailto:rfenner@gamecircus.com><br class=""><br \
class=""></blockquote><br class=""><blockquote type="cite" \
class="">______________________________________________________________________________<br \
class="">Member info:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit. \
hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.c \
om%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637 \
142711408966399&amp;sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balab \
it.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=02%7C01%7Cpeter.kokai%40oneidentit \
y.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C \
637142711408966399&amp;sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&amp;reserved=0</a><br \
class="">Documentation:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com \
%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&amp;data=02%7C01%7Cpeter.kokai \
%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec6066 \
03%7C0%7C0%7C637142711408966399&amp;sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&amp;data=02%7C01%7Cpeter.ko \
kai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec6 \
06603%7C0%7C0%7C637142711408966399&amp;sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&amp;reserved=0</a><br \
class="">FAQ:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com \
%2Fwiki%2Fsyslog-ng-faq&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de \
46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976 \
387&amp;sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
com%2Fwiki%2Fsyslog-ng-faq&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc \
9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408 \
976387&amp;sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&amp;reserved=0</a><br \
class=""><br class=""></blockquote>______________________________________________________________________________<br \
class="">Member info:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit. \
hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.c \
om%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637 \
142711408976387&amp;sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balab \
it.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=02%7C01%7Cpeter.kokai%40oneidentit \
y.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C \
637142711408976387&amp;sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&amp;reserved=0</a><br \
class="">Documentation:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com \
%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&amp;data=02%7C01%7Cpeter.kokai \
%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec6066 \
03%7C0%7C0%7C637142711408976387&amp;sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&amp;data=02%7C01%7Cpeter.ko \
kai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec6 \
06603%7C0%7C0%7C637142711408976387&amp;sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&amp;reserved=0</a><br \
class="">FAQ:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com \
%2Fwiki%2Fsyslog-ng-faq&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de \
46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976 \
387&amp;sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit. \
com%2Fwiki%2Fsyslog-ng-faq&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc \
9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408 \
976387&amp;sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&amp;reserved=0</a><br \
class=""><br class=""></blockquote><br \
class="">______________________________________________________________________________<br \
class="">Member info:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit. \
hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=02%7C01%7Cpeter.kokai%40oneidentity.c \
om%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637 \
142711408976387&amp;sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&amp;reserved=0" \
class="">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balab \
it.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=02%7C01%7Cpeter.kokai%40oneidentit \
y.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C \
637142711408976387&amp;sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&amp;reserved=0</a><br \
class="">Documentation:<span class="Apple-converted-space"> </span><a \
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com \
[Attachment #6 (text/plain)]
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic