'Re: [syslog-ng] Forwarding system startup messages'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       syslog-ng
Subject:    Re: [syslog-ng] Forwarding system startup messages
From:       Saurabh Shukla <saurabh () purestorage ! com>
Date:       2015-08-25 22:32:41
Message-ID: CAJfbNVa7JwJ2SggOB4dCwEqW5A94Qu+qGQ3TFGFHXitpjBKm6A () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

Done - https://github.com/balabit/syslog-ng/issues/659

-- Saurabh

On Tue, Aug 25, 2015 at 2:09 PM, Scheidler, Balázs <
balazs.scheidler@balabit.com> wrote:

> that seems like a good diagnosis. the dns resolution problem handling is a
> pretty recent one, so this must have fallen through the cracks.
>
> can you pls file a github ticket with your findings?
>
> thanks
>
>
> --
> Bazsi
>
> On Tue, Aug 25, 2015 at 8:31 PM, Saurabh Shukla <saurabh@purestorage.com>
> wrote:
>
>> Are you sure it's syslog-ng that writes /var/log/syslog ?
>>>
>> Yes. This is easy to verify. "file("/proc/kmsg"
>> program_override("kernel"));" directive adds a "kernel:" prefix to all
>> messages from the kernel and when I change this to something else, I see
>> the change in /var/log/syslog.
>>
>> I think the issue is that the output queue for network destinations is
>> created only if hostname resolution succeeds. During bootup, network
>> services are not up, so hostname resolution fails and no queue is created
>> for network destinations and hence syslog-ng fails to forward early startup
>> messages to network destinations.
>> This can be easily verified by bringing down the network, restarting
>> syslog-ng and then bringing up the network. Messages logged while the
>> network was down and after syslog-ng restart will not be forwarded to
>> network destinations.
>>
>> Ideally, I would assume that syslog-ng should unconditionally create
>> queues as soon as it reads destinations from configuration files. So the
>> questions now are is the current behavior intentional and can it be fixed?
>>
>> -- Saurabh
>>
>>
>> On Sun, Aug 23, 2015 at 11:11 PM, Scheidler, Balázs <
>> balazs.scheidler@balabit.com> wrote:
>>
>>> Are you sure it's syslog-ng that writes /var/log/syslog ?
>>>
>>> Sometimes early startup is handled by a different logger.
>>>
>>> Try to disable syslog-ng from starting up, boot the system and start
>>> syslog-ng manually. The kernel messages should be sitting in the dmesg
>>> buffer and syslog-ng should process them as soon as it starts.
>>>
>>> If it shows the same symptoms try to look at syslog-ng stats counters.
>>> Well you can do those even without the reboot game.
>>>
>>> $ syslog-ng-ctl stats
>>>
>>> Try to look for the processed counter for /proc/kmsg
>>> On Aug 24, 2015 3:48 AM, "Saurabh Shukla" <saurabh@purestorage.com>
>>> wrote:
>>>
>>>> I don't think clearing kernel buffers is an issue here since syslog-ng
>>>> is seeing the kernel messages during system boot up and logging them to
>>>> /var/log/syslog. However, it fails to forward them to the remote server. So
>>>> there is some issue with buffering messages for the remote destination.
>>>>
>>>> -- Saurabh
>>>>
>>>> On Sat, Aug 22, 2015 at 12:00 PM, Scheidler, Balázs <
>>>> balazs.scheidler@balabit.com> wrote:
>>>>
>>>>> Hmm. You don't even use /dev/kmsg or system, so this setup should
>>>>> work. Don't you happen to run anything that could read /proc/kmsg or clear
>>>>> the kernel ringbuffer behind the backs of syslog-ng?
>>>>> On Aug 22, 2015 8:11 PM, "Saurabh Shukla" <saurabh@purestorage.com>
>>>>> wrote:
>>>>>
>>>>>> Can you show your source declaration?
>>>>>>
>>>>>> I am using the syslog-ng.conf from here -
>>>>>> https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf
>>>>>>
>>>>>> Do you use systemd journal?
>>>>>>
>>>>>> No.
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 22, 2015 at 3:10 AM, Scheidler, Balázs <
>>>>>> balazs.scheidler@balabit.com> wrote:
>>>>>>
>>>>>>> Can you show your source declaration? Do you use systemd journal?
>>>>>>> On Aug 22, 2015 2:56 AM, "Saurabh Shukla" <saurabh@purestorage.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am running syslog-ng 3.6.4 and I have the following destination
>>>>>>>> and log path configured that forwards all messages to the destination:
>>>>>>>>
>>>>>>>> destination remote {
>>>>>>>>     network("remote.example.com" port(514) transport(tcp)
>>>>>>>> log_fifo_size(2048));
>>>>>>>> };
>>>>>>>> log { source(s_all); destination(remote); flags(flow-control);};
>>>>>>>>
>>>>>>>> When the system reboots, I see that startup messages from the
>>>>>>>> kernel are logged into /var/log/syslog.
>>>>>>>> syslog-ng establishes a connection to the remote destination around
>>>>>>>> 10 sec after the first message was logged into /var/log/syslog. However, it
>>>>>>>> fails to forward any message that was logged into /var/log/syslog during
>>>>>>>> the first 10 seconds even though I have the output buffer and flow control
>>>>>>>> configured.
>>>>>>>>
>>>>>>>> Is this a bug in syslog-ng or am I missing some configuration steps?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -- Saurabh
>>>>>>>>
>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>

[Attachment #5 (text/html)]

<div dir="ltr">Done -  <a \
href="https://github.com/balabit/syslog-ng/issues/659">https://github.com/balabit/syslog-ng/issues/659</a><div><br></div><div>-- \
Saurabh</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug \
25, 2015 at 2:09 PM, Scheidler, Balázs <span dir="ltr">&lt;<a \
href="mailto:balazs.scheidler@balabit.com" \
target="_blank">balazs.scheidler@balabit.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div>that seems like a good diagnosis. \
the dns resolution problem handling is a pretty recent one, so this must have fallen \
through the cracks. <br><br></div>can you pls file a github ticket with your \
findings?<br><br></div>thanks<span class="HOEnZb"><font \
color="#888888"><br><br></font></span></div><div class="gmail_extra"><span \
class="HOEnZb"><font color="#888888"><br clear="all"><div><div><div dir="ltr">-- \
<br>Bazsi<br></div></div></div></font></span><div><div class="h5"> <br><div \
class="gmail_quote">On Tue, Aug 25, 2015 at 8:31 PM, Saurabh Shukla <span \
dir="ltr">&lt;<a href="mailto:saurabh@purestorage.com" \
target="_blank">saurabh@purestorage.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><span><blockquote class="gmail_quote" \
style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Are \
you sure it&#39;s syslog-ng that writes /var/log/syslog \
?<br></blockquote></span><div>Yes. This is easy to verify. &quot;<span \
style="color:rgb(51,51,51);font-size:12px;line-height:16.7999992370605px;white-space:pre-wrap"><font \
face="monospace, monospace">file(&quot;/proc/kmsg&quot; \
program_override(&quot;kernel&quot;));</font></span>&quot; directive adds a \
&quot;<font face="monospace, monospace">kernel:</font>&quot; prefix to all messages \
from the kernel and when I change this to something else, I see the change in \
/var/log/syslog.</div><div><br></div><div>I think the issue is that the output queue \
for network destinations is created only if hostname resolution succeeds. During \
bootup, network services are not up, so hostname resolution fails and no queue is \
created for network destinations and hence syslog-ng fails to forward early startup \
messages to network destinations.</div><div>This can be easily verified by bringing \
down the network, restarting syslog-ng and then bringing up the network. Messages \
logged while the network was down and after syslog-ng restart will not be forwarded \
to network destinations.</div><div><br></div><div>Ideally, I would assume that \
syslog-ng should unconditionally create queues as soon as it reads destinations from \
configuration files. So the questions now are is the current behavior intentional and \
can it be fixed?</div><span><font color="#888888"><div><br></div><div>-- \
Saurabh<br></div></font></span><div><div><div class="gmail_extra"><br></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Sun, Aug 23, 2015 at 11:11 PM, \
Scheidler, Balázs <span dir="ltr">&lt;<a href="mailto:balazs.scheidler@balabit.com" \
target="_blank">balazs.scheidler@balabit.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Are you sure it&#39;s syslog-ng that writes \
/var/log/syslog ?</p> <p dir="ltr">Sometimes early startup is handled by a different \
logger.</p> <p dir="ltr">Try to disable syslog-ng from starting up, boot the system \
and start syslog-ng manually. The kernel messages should be sitting in the dmesg \
buffer and syslog-ng should process them as soon as it starts.</p> <p dir="ltr">If it \
shows the same symptoms try to look at syslog-ng stats counters. Well you can do \
those even without the reboot game.</p> <p dir="ltr">$ syslog-ng-ctl stats</p>
<p dir="ltr">Try to look for the processed counter for /proc/kmsg</p><div><div>
<div class="gmail_quote">On Aug 24, 2015 3:48 AM, &quot;Saurabh Shukla&quot; &lt;<a \
href="mailto:saurabh@purestorage.com" target="_blank">saurabh@purestorage.com</a>&gt; \
wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I don&#39;t think \
clearing kernel buffers is an issue here since syslog-ng is seeing the kernel \
messages during system boot up and logging them to /var/log/syslog. However, it fails \
to forward them to the remote server. So there is some issue with buffering messages \
for the remote destination.<div><br></div><div>-- Saurabh<br><div><br></div><div><div \
class="gmail_extra"><div class="gmail_quote">On Sat, Aug 22, 2015 at 12:00 PM, \
Scheidler, Balázs <span dir="ltr">&lt;<a href="mailto:balazs.scheidler@balabit.com" \
target="_blank">balazs.scheidler@balabit.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Hmm. You don&#39;t even use /dev/kmsg or system, \
so this setup should work. Don&#39;t you happen to run anything that could read \
/proc/kmsg or clear the kernel ringbuffer behind the backs of \
syslog-ng?</p><div><div> <div class="gmail_quote">On Aug 22, 2015 8:11 PM, \
&quot;Saurabh Shukla&quot; &lt;<a href="mailto:saurabh@purestorage.com" \
target="_blank">saurabh@purestorage.com</a>&gt; wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span \
style="font-size:12.8000001907349px">Can you show your source declaration?  \
</span></blockquote><div>I am using the syslog-ng.conf from here -  <a \
href="https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf" \
target="_blank">https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf</a></div><div><br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span \
style="font-size:12.8000001907349px">Do you use systemd \
journal?</span></blockquote><div>No.  </div><div><br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Sat, Aug 22, 2015 at 3:10 AM, \
Scheidler, Balázs <span dir="ltr">&lt;<a href="mailto:balazs.scheidler@balabit.com" \
target="_blank">balazs.scheidler@balabit.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Can you show your source declaration? Do you use \
systemd journal?</p> <div class="gmail_quote"><div><div>On Aug 22, 2015 2:56 AM, \
&quot;Saurabh Shukla&quot; &lt;<a href="mailto:saurabh@purestorage.com" \
target="_blank">saurabh@purestorage.com</a>&gt; wrote:<br \
type="attribution"></div></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">I am \
running syslog-ng 3.6.4 and I have the following destination and log path configured \
that forwards all messages to the destination:<div><div><br></div><div><font \
face="monospace, monospace">destination remote {</font></div><div><font \
face="monospace, monospace">      network(&quot;<a href="http://remote.example.com" \
target="_blank">remote.example.com</a>&quot; port(514) transport(tcp) \
log_fifo_size(2048));</font></div><div><font face="monospace, \
monospace">};</font></div><div><font face="monospace, monospace">log { source(s_all); \
destination(remote); \
flags(flow-control);};</font></div></div><div><br></div><div>When the system reboots, \
I see that startup messages from the kernel are logged into \
/var/log/syslog.</div><div>syslog-ng establishes a connection to the remote \
destination around 10 sec after the first message was logged into /var/log/syslog. \
However, it fails to forward any message that was logged into /var/log/syslog during \
the first 10 seconds even though I have the output buffer and flow control \
configured.<br></div><div><br></div><div>Is this a bug in syslog-ng or am I missing \
some configuration steps?</div><div><br></div><div>Thanks,</div><div>-- \
Saurabh</div></div> <br></div></div>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div>
<br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div><br></div>
<br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div>
</div></div><br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div>
</div></div><br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div><br></div></div></div>
<br>______________________________________________________________________________<br>
 Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
<br></blockquote></div><br></div>

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

[prev in list] [next in list] [prev in thread] [next in thread] 