[prev in list] [next in list] [prev in thread] [next in thread]
List: ossec-list
Subject: [ossec-list] Re: ERROR: Unable to send file 'merged.mg' to agent.
From: Graeme Stewart <gstewart () gmail ! com>
Date: 2016-07-29 21:56:08
Message-ID: b35e79ba-3305-44c6-b8f9-2f9bebbe3fdf () googlegroups ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Awesome! Many thanks, this is exactly what I was looking for.
On Friday, July 29, 2016 at 12:16:35 PM UTC-7, Victor Fernandez wrote:
>
> Hi Graeme.
>
> I agree, it would be great to print on the log that the agent became
> disconnected. The SEC_ERROR definition is shared between manager and
> agents, but it's possible to extend some other messages. In fact, the line
> at sendmsg.c that tests if the agent is disconnected (more than 20
> minutes since the last keep-alive) is the only one that doesn't log an
> error.
>
> I did some modifications at the Wazuh repository, maybe it's useful to you:
>
>
> https://github.com/wazuh/ossec-wazuh/commit/efbd5c17cc3ea5109ea978208b11da6c98fa3083
>
> See below an example of the new log format for the error:
>
> 2016/07/29 11:43:57 ossec-remoted(1245): ERROR: Sending message to
> disconnected agent '001'.
> 2016/07/29 11:43:57 ossec-remoted(1217): ERROR: Error creating encrypted
> message.
> 2016/07/29 11:43:57 ossec-remoted(1246): ERROR: Unable to send file '
> merged.mg' to agent '001' (centos).
>
> I hope this leads you to find the problem.
>
> Kind regards.
>
>
> On Friday, July 29, 2016 at 8:19:56 AM UTC-7, Graeme Stewart wrote:
> >
> > Hi Victor,
> >
> > Huge thanks for the detail, this would explain exactly why we're seeing
> > this; our OSSEC managers are likely overloaded.
> >
> > It would be very helpful to include the agentid in the logfile to
> > understand / track where this is occurring and the number of unique agents
> > that are impacted, perhaps something like:
> >
> > From: src/error_messages/error_messages.h
> > #define SEC_ERROR "%s(1217): ERROR: Error creating encrypted
> > message for: '%s')."
> >
> > Then inside: src/remoted/sendmsg.c
> > msg_size = CreateSecMSG(&keys, msg, crypt_msg, agentid);
> > if (msg_size == 0) {
> > merror(SEC_ERROR, ARGV0, agentid);
> > return (-1);
> > }
> >
> > The clustered nature of this issue leads me to suspect it's repeating
> > this error in the logfiles multiple times for a connection attempt across
> > only one or two agents.
> >
> > Again, many thanks for the detailed response.
> >
> > Graeme
> >
> > On Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote:
> > >
> > > Hi Graeme.
> > >
> > > According to the log, I think the problem occurs when the manager tries
> > > to send the merged.mg to an agent that has not sent the keep-alive in
> > > the last 20 minutes. This may happen if a lot of agents get connected, or
> > > send the keep-alive at the same time.
> > >
> > > So, if many agents send a keep-alive, the manager takes more than 20
> > > minutes to send the merged.mg to an agent, and that agent hasn't sent
> > > the keep-alive again, this problem occurs.
> > >
> > > I did some math: the manager sleeps one second every time it sends 27
> > > KB. With a 150 KB merged.mg, OSSEC takes 20 minutes to send the
> > > complete file to about 216 agents.
> > >
> > > The 20-minutes check appears on src/remoted/sendmsg.c:
> > >
> > > /* If we don't have the agent id, ignore it */
> > > if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) {
> > > return (-1);
> > > }
> > >
> > > NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an
> > > agent as disconnected when it hasn't send the keep-alive in the last 30:30
> > > minutes, as we can see at src/shared/read-agents.c:
> > >
> > > if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) {
> > > return (GA_STATUS_ACTIVE);
> > > }
> > >
> > > Because of this, I think that this may be an issue.
> > >
> > > I think that a good approach would be to check that there aren't alerts
> > > about disconnected agents that connected recently.
> > >
> > > Kind regards.
> > >
> > >
> > > On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote:
> > > >
> > > > Seeing a lot of errors in the logfiles like this:
> > > >
> > > > 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file '
> > > > merged.mg' to agent.
> > > > 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating
> > > > encrypted message.
> > > > 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file '
> > > > merged.mg' to agent.
> > > > 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating
> > > > encrypted message.
> > > > 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file '
> > > > merged.mg' to agent.
> > > > 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating
> > > > encrypted message.
> > > > 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file '
> > > > merged.mg' to agent.
> > > > 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating
> > > > encrypted message.
> > > > 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file '
> > > > merged.mg' to agent.
> > > > 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating
> > > > encrypted message.
> > > > 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file '
> > > > merged.mg' to agent.
> > > > 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating
> > > > encrypted message.
> > > >
> > > > Any guidance on troubleshooting? Search hasn't turned up much other
> > > > than delete merged.mg and restart (which we've tried to no success)...
> > > >
> > >
--
---
You received this message because you are subscribed to the Google Groups \
"ossec-list" group. To unsubscribe from this group and stop receiving emails from it, \
send an email to ossec-list+unsubscribe@googlegroups.com. For more options, visit \
https://groups.google.com/d/optout.
[Attachment #5 (text/html)]
<div dir="ltr">Awesome! Many thanks, this is exactly what I was looking \
for.<br><br>On Friday, July 29, 2016 at 12:16:35 PM UTC-7, Victor Fernandez \
wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: \
0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr">Hi \
Graeme.<div><br></div><div>I agree, it would be great to print on the log that the \
agent became disconnected. The <span style="font-family:"courier \
new",monospace">SEC_ERROR</span><font face="arial, sans-serif"> definition is \
shared between manager and agents, but it's possible to extend some other \
messages. In fact, the line at </font><font face="courier new, \
monospace">sendmsg.c</font><font face="arial, sans-serif"> that tests if the agent is \
disconnected (more than 20 minutes since the last keep-alive) is the only one that \
doesn't log an error.</font></div><div><font face="arial, \
sans-serif"><br></font></div><div><font face="arial, sans-serif">I did some \
modifications at the Wazuh repository, maybe it's useful to \
you:</font></div><div><font face="arial, sans-serif"><br></font></div><div><font \
face="arial, sans-serif"><a \
href="https://github.com/wazuh/ossec-wazuh/commit/efbd5c17cc3ea5109ea978208b11da6c98fa3083" \
target="_blank" rel="nofollow" \
onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2 \
Fwazuh%2Fossec-wazuh%2Fcommit%2Fefbd5c17cc3ea5109ea978208b11da6c98fa3083\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbNFA7DQS7NR_s7E2axpBudryaHg';return \
true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.co \
m%2Fwazuh%2Fossec-wazuh%2Fcommit%2Fefbd5c17cc3ea5109ea978208b11da6c98fa3083\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbNFA7DQS7NR_s7E2axpBudryaHg';return \
true;">https://github.com/wazuh/<wbr>ossec-wazuh/commit/<wbr>efbd5c17cc3ea5109ea978208b11da<wbr>6c98fa3083</a></font></div><div><br></div><div><font \
face="arial, sans-serif">See below an example of the new log format for the \
error:</font></div><div><font face="arial, \
sans-serif"><br></font></div><div><div><font face="courier new, monospace">2016/07/29 \
11:43:57 ossec-remoted(1245): ERROR: Sending message to disconnected agent \
'001'.</font></div><div><font face="courier new, monospace">2016/07/29 \
11:43:57 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div><div><font face="courier new, monospace">2016/07/29 11:43:57 \
ossec-remoted(1246): ERROR: Unable to send file '<a href="http://merged.mg" \
target="_blank" rel="nofollow" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent '001' \
(centos).</font></div></div><div><br></div><div><font face="arial, sans-serif">I hope \
this leads you to find the problem.</font></div><div><font face="arial, \
sans-serif"><br></font></div><div><font face="arial, sans-serif">Kind \
regards.</font></div><div><font face="arial, sans-serif"><br></font><br>On Friday, \
July 29, 2016 at 8:19:56 AM UTC-7, Graeme Stewart wrote:<blockquote \
class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Victor,<div><br></div><div>Huge thanks for \
the detail, this would explain exactly why we're seeing this; our OSSEC managers \
are likely overloaded.</div><div><br></div><div>It would be very helpful to include \
the agentid in the logfile to understand / track where this is occurring and the \
number of unique agents that are impacted, perhaps something \
like:</div><div><br></div><div>From: <font face="courier new, \
monospace">src/error_messages/<wbr>error_messages.h</font></div><div><font \
face="courier new, monospace">#define SEC_ERROR "%s(1217): ERROR: Error \
creating encrypted message for: \
'%s')."</font><br></div><div><br></div><div>Then inside:<font \
face="courier new, monospace"> src/remoted/sendmsg.c</font></div><div><font \
face="courier new, monospace">msg_size = CreateSecMSG(&keys, msg, crypt_msg, \
agentid);</font></div><div><div><font face="courier new, monospace">if (msg_size == \
0) {</font></div><div><font face="courier new, monospace"> merror(SEC_ERROR, \
ARGV0, agentid);</font></div><div><font face="courier new, monospace"> return \
(-1);</font></div><div><font face="courier new, \
monospace">}</font></div><div><br></div><div>The clustered nature of this issue leads \
me to suspect it's repeating this error in the logfiles multiple times for a \
connection attempt across only one or two agents.</div><div><br></div><div>Again, \
many thanks for the detailed response.</div><div><br></div><div>Graeme</div><br>On \
Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote:<blockquote \
class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Graeme.<div><br></div><div>According to the \
log, I think the problem occurs when the manager tries to send the <font \
face="courier new, monospace"><a href="http://merged.mg" rel="nofollow" \
target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F% \
2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a></font> to an agent that has not sent the keep-alive in the last \
20 minutes. This may happen if a lot of agents get connected, or send the keep-alive \
at the same time. </div><div><br></div><div>So, if many agents send a keep-alive, \
the manager takes more than 20 minutes to send the <font face="courier new, \
monospace"><a href="http://merged.mg" rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a></font> to an agent, and that agent hasn't sent the \
keep-alive again, this problem occurs.</div><div><br></div><div>I did some math: the \
manager sleeps one second every time it sends 27 KB. With a 150 KB <font \
face="courier new, monospace"><a href="http://merged.mg" rel="nofollow" \
target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F% \
2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a></font>, OSSEC takes 20 minutes to send the complete file to \
about 216 agents.</div><div><br></div><div>The 20-minutes check appears on <font \
face="courier new, monospace">src/remoted/sendmsg.</font>c:</div><div><br></div><div><div><font \
face="courier new, monospace">/* If we don't have the agent id, ignore it \
*/</font></div><div><font face="courier new, monospace">if \
(keys.keyentries[agentid]-><wbr>rcvd < (time(0) - (2 * NOTIFY_TIME))) \
{</font></div><div><font face="courier new, monospace"> return \
(-1);</font></div><div><font face="courier new, \
monospace">}</font></div></div><div><br></div><div>NOTIFY_TIME is 600 (10 minutes) by \
default. Nevertheless OSSEC labels an agent as disconnected when it hasn't send \
the keep-alive in the last 30:30 minutes, as we can see at <font face="courier new, \
monospace">src/shared/read-agents.c</font>:</div><div><br></div><div><div><font \
face="courier new, monospace">if (file_status.st_mtime > (time(0) - (3 * \
NOTIFY_TIME + 30))) {</font></div><div><font face="courier new, monospace"> \
return (GA_STATUS_ACTIVE);</font></div><div><font face="courier new, \
monospace">}</font></div></div><div><br></div><div>Because of this, I think that this \
may be an issue.</div><div><br></div><div>I think that a good approach would be to \
check that there aren't alerts about disconnected agents that connected \
recently.</div><div><br></div><div>Kind \
regards.</div><div><br></div><div><br></div><div>On Thursday, July 28, 2016 at \
9:43:32 AM UTC-7, Graeme Stewart wrote:<blockquote class="gmail_quote" \
style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
dir="ltr">Seeing a lot of errors in the logfiles like this:<div><font face="courier \
new, monospace"><br></font></div><div><div><font face="courier new, \
monospace">2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file '<a \
href="http://merged.mg" rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent.</font></div><div><font face="courier new, \
monospace">2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div><div><font face="courier new, monospace">2016/07/28 16:41:50 \
ossec-remoted: ERROR: Unable to send file '<a href="http://merged.mg" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent.</font></div><div><font face="courier new, \
monospace">2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div><div><font face="courier new, monospace">2016/07/28 16:41:50 \
ossec-remoted: ERROR: Unable to send file '<a href="http://merged.mg" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent.</font></div><div><font face="courier new, \
monospace">2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div><div><font face="courier new, monospace">2016/07/28 16:41:52 \
ossec-remoted: ERROR: Unable to send file '<a href="http://merged.mg" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent.</font></div><div><font face="courier new, \
monospace">2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div><div><font face="courier new, monospace">2016/07/28 16:41:52 \
ossec-remoted: ERROR: Unable to send file '<a href="http://merged.mg" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent.</font></div><div><font face="courier new, \
monospace">2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div><div><font face="courier new, monospace">2016/07/28 16:41:54 \
ossec-remoted: ERROR: Unable to send file '<a href="http://merged.mg" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a>' to agent.</font></div><div><font face="courier new, \
monospace">2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted \
message.</font></div></div><div><br></div><div>Any guidance on troubleshooting? \
Search hasn't turned up much other than delete <a href="http://merged.mg" \
rel="nofollow" target="_blank" \
onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fmerged.mg\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFfDgVaaQO-9bcM51AgaUAG47JpHA';return \
true;">merged.mg</a> and restart (which we've tried to no \
success)...<br></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div>
<p></p>
-- <br />
<br />
--- <br />
You received this message because you are subscribed to the Google Groups \
"ossec-list" group.<br /> To unsubscribe from this group and stop receiving \
emails from it, send an email to <a \
href="mailto:ossec-list+unsubscribe@googlegroups.com">ossec-list+unsubscribe@googlegroups.com</a>.<br \
/> For more options, visit <a \
href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br \
/>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic