[prev in list] [next in list] [prev in thread] [next in thread]
List: opennms-buglist
Subject: [Buglist] [JIRA] (NMS-8376) Poller threads can get locked up while waiting for outages to be created
From: "Jeff Gehlbach (JIRA)" <jira () opennms ! org>
Date: 2016-04-27 13:53:21
Message-ID: JIRA.25872.1461763561000.5731.1461765201479 () Atlassian ! JIRA
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (text/html)]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" \
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html \
xmlns="http://www.w3.org/1999/xhtml"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, \
maximum-scale=1.0" /> <base href="http://issues.opennms.org" /> <title>Message \
Title</title> </head>
<body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: \
14px; line-height: 1.429"> <table id="background-table" cellpadding="0" \
cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; \
mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <!-- header here -->
<tr>
<td id="header-pattern-container" style="padding: 0px; \
border-collapse: collapse; padding: 10px 20px"> <table id="header-pattern" \
cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<td id="header-avatar-image-container" valign="top" \
style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; \
padding-right: 8px"> <img id="header-avatar-image" class="image_fix" \
src="cid:jira-generated-image-avatar-jeffg-292e09f6-83dd-485d-b55f-c7dd58a2f37f" \
height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" /> \
</td> <td id="header-text-container" valign="middle" style="padding: 0px; \
border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; \
font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: \
1px"> <a class="user-hover" rel="jeffg" id="email_jeffg" \
href="http://issues.opennms.org/secure/ViewProfile.jspa?name=jeffg" \
style="color:#005e00;; color: #3b73af; text-decoration: none">Jeff Gehlbach</a> \
<strong>updated</strong> an issue </td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; \
border-collapse: collapse; padding: 0 20px"> <table id="email-content-table" \
cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: \
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; \
border-collapse: separate"> <tr>
<!-- there needs to be content in the cell for it to \
render in some clients --> <td class="email-content-rounded-top mobile-expand" \
style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; \
height: 15px; background-color: #fff; border-left: 1px solid #ccc; border-top: 1px \
solid #ccc; border-right: 1px solid #ccc; border-bottom: 0; border-top-right-radius: \
5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 \
16px; mso-line-height-rule: exactly">
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " \
style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; \
border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 \
16px; background-color: #fff"> <table class="page-title-pattern" cellspacing="0" \
cellpadding="0" border="0" width="100%" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<td class="page-title-pattern-first-line " \
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; \
font-size: 14px; padding-top: 10px"> <a href="http://issues.opennms.org/browse/NMS" \
style="color: #3b73af; text-decoration: none">OpenNMS</a> / <a \
href="http://issues.opennms.org/browse/NMS-8376" style="color: #3b73af; \
text-decoration: none"><img \
src="cid:jira-generated-image-static-bug-869d7b76-9791-4749-97d0-afdcdc3cced5" \
height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: \
text-bottom" /></a> <a href="http://issues.opennms.org/browse/NMS-8376" style="color: \
#3b73af; text-decoration: none">NMS-8376</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: \
0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: \
30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> \
<span class="page-title-pattern-header" style="font-family: Arial, sans-serif; \
padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; \
mso-line-height-rule: exactly; vertical-align: middle"> <a \
href="http://issues.opennms.org/browse/NMS-8376" style="color: #3b73af; \
text-decoration: none">Poller threads can get locked up while waiting for outages to \
be created/resolved</a> </span> </td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand \
wrapper-special-margin" style="padding: 0px; border-collapse: collapse; border-left: \
1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; \
padding: 0 15px 0 16px; background-color: #fff; padding-top: 10px; padding-bottom: \
5px">
<table class="keyvalue-table" style="border-collapse: \
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<th style="color: #707070; font: normal \
14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0"> \
Change By: </th>
<td style="padding: 0px; border-collapse: \
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; \
vertical-align: top"> <a class="user-hover" rel="jeffg" id="email_jeffg" \
href="http://issues.opennms.org/secure/ViewProfile.jspa?name=jeffg" \
style="color:#005e00;; color: #3b73af; text-decoration: none">Jeff Gehlbach</a> \
</td> </tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand \
issue-description-container" style="padding: 0px; border-collapse: collapse; \
border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; \
border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 5px; \
padding-bottom: 10px"> <table class="text-paragraph-pattern" cellspacing="0" \
cellpadding="0" border="0" width="100%" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; \
font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: \
2px"> <tr>
<td class="text-paragraph-pattern-container \
mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 \
10px 0"> <span class="diffcontext">The segments of the following&n \
bsp;stack trace show that several poller threads ar \
e locked, waiting to append an event to a  \
;linked list, while another thread, which has a&nbs \
p;lock on the linked list in processing the outages.<br \
/><br />Ideally, the poller threads should not need to block and wait for this lock.<br \
/><br />{noformat}<br />24 threads with this stack:<br \
/>"Poller-Thread-10-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-16-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-17-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-18-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-20-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-21-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-22-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-27-of-40": waiting to acquire [0x0000000707f46848]</span> \
<span class="diffaddedchars" style="background-color:#ddfade;">24</span> <span \
class="diffcontext"><br \
/>"Poller-Thread-28-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-29-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-3-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-30-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-31-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-33-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-35-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-36-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-37-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-39-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-4-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-40-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-5-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-6-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-8-of-40": waiting to acquire [0x0000000707f46848]<br \
/>"Poller-Thread-9-of-40": waiting to acquire [0x0000000707f46848]<br \
/> at org.opennms.netmgt.poller.DefaultPollContext.sendEvent(DefaultPollContext.java:234)<br \
/> at org.opennms.netmgt.poller.pollables.PollableElement.processGoingDown(PollableElement.java:407)<br \
/>{noformat}<br /><br />{noformat}<br \
/>"OpenNMS.Poller.DefaultPollContext-Thread": running, holding [0x0000000729a94c98, 0x0000000729bd5bc0, 0x0000000707f46848]<br \
/> at java.net.SocketInputStream.socketRead0(Native Method)<br /> \
at java.net.SocketInputStream.read(SocketInputStream.java:152)<br /> \
at java.net.SocketInputStream.read(SocketInputStream.java:122)<br /> \
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)<br \
/> at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)<br \
/> at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)<br \
/> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)<br /> \
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1704)<br \
/> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)<br \
/> at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560)<br \
/> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)<br \
/> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)<br \
/> at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)<br \
/> at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46)<br \
/> at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2591)<br \
/> at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495)<br \
/> at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822)<br \
/> at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)<br \
/> at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)<br /> \
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)<br /> \
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)<br /> \
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)<br \
/> at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)<br \
/> at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)<br /> \
at org.springframework.orm.hibernate3.HibernateAccessor.flushIfNecessary(HibernateAccessor.java:390)<br \
/> at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:408)<br \
/> at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:375)<br \
/> at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:738)<br \
/> at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.saveOrUpdate(AbstractDaoHibernate.java:410)<br \
/> at org.opennms.netmgt.poller.QueryManagerDaoImpl.updateResolvedOutageWithEventId(QueryManagerDaoImpl.java:175)<br \
/> at org.opennms.netmgt.poller.DefaultPollContext$2.run(DefaultPollContext.java:351)<br \
/> at org.opennms.netmgt.poller.pollables.PendingPollEvent.processPending(PendingPollEvent.java:145)<br \
/> at org.opennms.netmgt.poller.DefaultPollContext.onEvent(DefaultPollContext.java:408)<br \
/> at org.opennms.netmgt.eventd.EventIpcManagerDefaultImpl$EventListenerExecutor$2.run(EventIpcManagerDefaultImpl.java:176)<br \
/>{noformat}</span> </td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " \
style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; \
border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 \
16px; background-color: #fff"> <table id="actions-pattern" cellspacing="0" \
cellpadding="0" border="0" width="100%" style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; \
font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: \
1px"> <tr>
<td id="actions-pattern-container" \
valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px \
24px; vertical-align: middle; padding-left: 0">
<table align="left" \
style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> \
<tr> <td class="actions-pattern-action-icon-container" style="padding: 0px; \
border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; \
line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; \
vertical-align: middle"> <a \
href="http://issues.opennms.org/browse/NMS-8376#add-comment" target="_blank" \
title="Add Comment" style="color: #3b73af; text-decoration: none"> <img \
class="actions-pattern-action-icon-image" \
src="cid:jira-generated-image-static-comment-icon-2e3dbcbd-d01b-47c1-b5e1-72f8a8bef0a2" \
alt="Add Comment" title="Add Comment" height="16" width="16" border="0" \
style="vertical-align: middle" /> </a> </td>
<td \
class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: \
collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; \
mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a \
href="http://issues.opennms.org/browse/NMS-8376#add-comment" target="_blank" \
title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> \
</td> </tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- there needs to be content in the cell for it to render \
in some clients --> <tr>
<td class="email-content-rounded-bottom mobile-expand" \
style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; \
height: 5px; line-height: 5px; background-color: #fff; border-top: 0; border-left: \
1px solid #ccc; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; \
border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; \
mso-line-height-rule: exactly">
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: \
collapse; padding: 12px 20px"> <table id="footer-pattern-container" cellspacing="0" \
cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; \
mso-table-rspace: 0pt"> <tr>
<td id="footer-pattern-text" class="mobile-resize-text" \
width="100%" style="padding: 0px; border-collapse: collapse; color: #999; font-size: \
12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: \
exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span \
id="footer-build-information">(v6.3.15#6346-<span \
title="dbc023dd75cecacf443c4b235f66124b15f5c5fe" \
data-commit-id="dbc023dd75cecacf443c4b235f66124b15f5c5fe}">sha1:dbc023d</span>)</span> \
</td>
<td id="footer-pattern-logo-desktop-container" \
valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; \
vertical-align: top">
<table style="border-collapse: collapse; \
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> <tr>
<td id="footer-pattern-logo-desktop-padding" \
style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img \
id="footer-pattern-logo-desktop" \
src="cid:jira-generated-image-static-footer-desktop-logo-ac27b466-b431-43b9-8994-e3b88dc9ed04" \
alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" \
/> </td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>
[Attachment #5 (image/png)]
[Attachment #6 (image/png)]
[Attachment #7 (image/png)]
[Attachment #8 (image/png)]
[Attachment #9 (text/plain)]
_______________________________________________
Buglist mailing list
Buglist@lists.opennms.com
http://lists.opennms.com/mailman/listinfo/buglist
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic