[prev in list] [next in list] [prev in thread] [next in thread]
List: opennms-buglist
Subject: [Buglist] [JIRA] (NMS-8777) Incoming syslog/trap flood can overwhelm new handler code
From: "Seth Leger (JIRA)" <jira () opennms ! org>
Date: 2016-09-29 19:07:43
Message-ID: JIRA.26966.1475175966000.1033.1475176063292 () Atlassian ! JIRA
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (text/html)]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" \
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html \
xmlns="http://www.w3.org/1999/xhtml"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, \
maximum-scale=1.0" /> <base href="https://issues.opennms.org" /> <title>Message \
Title</title> </head>
<body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: \
14px; line-height: 1.429"> <table id="background-table" cellpadding="0" \
cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; \
mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <!-- header here -->
<tr>
<td id="header-pattern-container" style="padding: 0px; \
border-collapse: collapse; padding: 10px 20px"> <table id="header-pattern" \
cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<td id="header-avatar-image-container" valign="top" \
style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; \
padding-right: 8px"> <img id="header-avatar-image" class="image_fix" \
src="https://secure.gravatar.com/avatar/4473de6c2a06796f7cfdf7a89c437a24?d=mm&s=48" \
height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" /> \
</td> <td id="header-text-container" valign="middle" style="padding: 0px; \
border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; \
font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: \
1px"> <a class="user-hover" rel="seth" id="email_seth" \
href="https://issues.opennms.org/secure/ViewProfile.jspa?name=seth" \
style="color:#005e00;; color: #3b73af; text-decoration: none">Seth Leger</a> \
<strong>created</strong> an issue </td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; \
border-collapse: collapse; padding: 0 20px"> <table id="email-content-table" \
cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: \
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; \
border-collapse: separate"> <tr>
<!-- there needs to be content in the cell for it to \
render in some clients --> <td class="email-content-rounded-top mobile-expand" \
style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; \
height: 15px; background-color: #fff; border-left: 1px solid #ccc; border-top: 1px \
solid #ccc; border-right: 1px solid #ccc; border-bottom: 0; border-top-right-radius: \
5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 \
16px; mso-line-height-rule: exactly">
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " \
style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; \
border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 \
16px; background-color: #fff"> <table class="page-title-pattern" cellspacing="0" \
cellpadding="0" border="0" width="100%" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<td class="page-title-pattern-first-line " \
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; \
font-size: 14px; padding-top: 10px"> <a href="https://issues.opennms.org/browse/NMS" \
style="color: #3b73af; text-decoration: none">OpenNMS</a> / <a \
href="https://issues.opennms.org/browse/NMS-8777" style="color: #3b73af; \
text-decoration: none"><img \
src="cid:jira-generated-image-static-bug-0c42c3ff-c982-4f2b-8f22-7dd0522c6ca8" \
height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: \
text-bottom" /></a> <a href="https://issues.opennms.org/browse/NMS-8777" \
style="color: #3b73af; text-decoration: none">NMS-8777</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: \
0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: \
30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> \
<span class="page-title-pattern-header" style="font-family: Arial, sans-serif; \
padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; \
mso-line-height-rule: exactly; vertical-align: middle"> <a \
href="https://issues.opennms.org/browse/NMS-8777" style="color: #3b73af; \
text-decoration: none">Incoming syslog/trap flood can overwhelm new handler code</a> \
</span> </td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand \
wrapper-special-margin" style="padding: 0px; border-collapse: collapse; border-left: \
1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; \
padding: 0 15px 0 16px; background-color: #fff; padding-top: 10px; padding-bottom: \
5px">
<table class="keyvalue-table" style="border-collapse: \
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Issue Type: </th>
<td class="has-icon" style="padding: 0px; \
border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 \
2px 5px; vertical-align: top"> <img \
src="cid:jira-generated-image-static-bug-0c42c3ff-c982-4f2b-8f22-7dd0522c6ca8" \
height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: \
text-bottom" /> Bug </td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Affects Versions: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top"> 19.0.0
</td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Assignee: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top"> Unassigned
</td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Components: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top">
Event Reception - SNMP Traps, Event \
Reception - Syslog </td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Created: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top"> 29/Sep/16 3:06 PM
</td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Fix Versions: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top"> 19.0.0
</td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Priority: </th>
<td class="has-icon" style="padding: 0px; \
border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 \
2px 5px; vertical-align: top"> <img \
src="cid:jira-generated-image-static-blocker-ccc2ab1b-d869-440a-828b-c75badf08aa6" \
height="16" width="16" border="0" align="absmiddle" alt="Blocker" \
style="vertical-align: text-bottom" /> Blocker </td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Reporter: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top"> <a class="user-hover" rel="seth" id="email_seth" \
href="https://issues.opennms.org/secure/ViewProfile.jspa?name=seth" \
style="color:#005e00;; color: #3b73af; text-decoration: none">Seth Leger</a> </td>
</tr>
<tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Security Level: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top">
Default (Default Security Scheme)
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand \
issue-description-container" style="padding: 0px; border-collapse: collapse; \
border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; \
border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 5px; \
padding-bottom: 10px"> <table class="text-paragraph-pattern" cellspacing="0" \
cellpadding="0" border="0" width="100%" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; \
font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: \
2px"> <tr>
<td class="text-paragraph-pattern-container \
mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 \
10px 0"> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">If a large \
backlog of syslog or trap messages generated by a Minion is waiting on a Kafka broker \
and OpenNMS is started up, it will attempt to stream all of the messages at once into \
the Camel messaging system. This will exhaust all of the Java heap space and lead to \
an OutOfMemoryError if the number of messages in the backlog is too large.</p> <p \
style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">To provide back-pressure \
on this queue and prevent memory from being exhausted, we should make the incoming \
Camel SEDA queue have a limited size and mark it as "blockWhenFull".</p> \
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">It will normally be \
very close to empty because events will be processed quickly after they are received \
so putting a limit on the queue size should only come into play when there is a \
significant backlog of messages that has accumulated in the messaging channel.</p> \
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">Since most syslog and \
trap messages will be 1KB - 2KB in size, I would recommend a default queue size of \
50,000 which should consume roughly 100MB of RAM under full load.</p> \
<p \
style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">This needs to be done in \
4 contexts:</p> <ul>
<li>
\
blueprint-syslog-handler-default.xml </li>
<li>
\
blueprint-syslog-handler-kafka-default.xml </li>
<li>
\
blueprint-trapd-handler-default.xml </li>
<li>
\
blueprint-trapd-handler-kafka-default.xml </li>
</ul>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " \
style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; \
border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 \
16px; background-color: #fff"> <table id="actions-pattern" cellspacing="0" \
cellpadding="0" border="0" width="100%" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; \
font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: \
1px"> <tr>
<td id="actions-pattern-container" \
valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px \
24px; vertical-align: middle; padding-left: 0">
<table align="left" \
style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> \
<tr> <td class="actions-pattern-action-icon-container" style="padding: 0px; \
border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; \
line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; \
vertical-align: middle"> <a \
href="https://issues.opennms.org/browse/NMS-8777#add-comment" target="_blank" \
title="Add Comment" style="color: #3b73af; text-decoration: none"> <img \
class="actions-pattern-action-icon-image" \
src="cid:jira-generated-image-static-comment-icon-59635098-9f7e-4df8-8b8f-b9f1368b4104" \
alt="Add Comment" title="Add Comment" height="16" width="16" border="0" \
style="vertical-align: middle" /> </a> </td>
<td \
class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: \
collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; \
mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a \
href="https://issues.opennms.org/browse/NMS-8777#add-comment" target="_blank" \
title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> \
</td> </tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- there needs to be content in the cell for it to render \
in some clients --> <tr>
<td class="email-content-rounded-bottom mobile-expand" \
style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; \
height: 5px; line-height: 5px; background-color: #fff; border-top: 0; border-left: \
1px solid #ccc; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; \
border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; \
mso-line-height-rule: exactly">
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: \
collapse; padding: 12px 20px"> <table id="footer-pattern-container" cellspacing="0" \
cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; \
mso-table-rspace: 0pt"> <tr>
<td id="footer-pattern-text" class="mobile-resize-text" \
width="100%" style="padding: 0px; border-collapse: collapse; color: #999; font-size: \
12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: \
exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span \
id="footer-build-information">(v6.3.15#6346-<span \
title="dbc023dd75cecacf443c4b235f66124b15f5c5fe" \
data-commit-id="dbc023dd75cecacf443c4b235f66124b15f5c5fe}">sha1:dbc023d</span>)</span> \
</td>
<td id="footer-pattern-logo-desktop-container" \
valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; \
vertical-align: top">
<table style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<td id="footer-pattern-logo-desktop-padding" \
style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img \
id="footer-pattern-logo-desktop" \
src="cid:jira-generated-image-static-footer-desktop-logo-ef6f31ed-1b15-436b-b727-96d740d51d43" \
alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" \
/> </td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>
[Attachment #5 (image/png)]
[Attachment #6 (image/png)]
[Attachment #7 (image/png)]
[Attachment #8 (image/png)]
[Attachment #9 (text/plain)]
_______________________________________________
Buglist mailing list
Buglist@lists.opennms.com
http://lists.opennms.com/mailman/listinfo/buglist
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic