[prev in list] [next in list] [prev in thread] [next in thread] 

List:       spacewalk-list
Subject:    [Spacewalk-list] Errors cancelling scheduled task with many clients
From:       Ian Bishop <IBISHOP () BCLC ! com>
Date:       2016-10-31 17:13:18
Message-ID: F81933E375C3934B83C68D52ADAE7D4AAA291A6A () kam1opeml03
[Download RAW message or body]

Hi, I am trying to cancel a scheduled task which has about 1246 clients listed.  I am \
able to cancel scheduled tasks with fewer clients.

Here is what I find in the logs:

==> /var/log/rhn/rhn_web_api.log <==
[2016-10-31 10:06:33,126] ERROR - REQUESTED FROM: 10.20.0.60 CALL: \
                schedule.cancelActions(xxxxxxx, [64272]) CALLER: (xxxxxxx) TIME: \
                10.398 seconds
redstone.xmlrpc.XmlRpcFault: unhandled internal exception: could not execute query
        at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:208)
        at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
        at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
                
        at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
                
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
                
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
                
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:100)
                
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:57)
                
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
                
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
                
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
                
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
                
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
                
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
                
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
       at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
                
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
                
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.lang.Thread.run(Thread.java:745)

==> /var/log/tomcat/catalina.2016-10-31.log <==
Oct 31, 2016 10:06:33 AM redstone.xmlrpc.XmlRpcDispatcher writeError
WARNING: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: could not execute \
query

I have tried turning up the verbosity in rhn.conf and restarting the services.  Our \
DBA says it doesn't look like the query is getting to the database.  I get the same \
error through the website and spacecmd.

Any ideas?

Thanks, Ian

________________________________
This email is intended only for the addressee. It may contain confidential or \
proprietary information that cannot be disclosed without BCLC's permission. If you \
have received this email in error, please notify the sender immediately and delete \
the email.


[Attachment #3 (text/html)]

<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi, I am trying to cancel a scheduled task which has about 1246 \
clients listed.&nbsp; I am able to cancel scheduled tasks with fewer \
clients.<o:p></o:p></p> <p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Here is what I find in the logs:<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">==&gt; /var/log/rhn/rhn_web_api.log &lt;==<o:p></o:p></p>
<p class="MsoNormal">[2016-10-31 10:06:33,126] ERROR - REQUESTED FROM: 10.20.0.60 \
CALL: schedule.cancelActions(xxxxxxx, [64272]) CALLER: (xxxxxxx) TIME: 10.398 \
seconds<o:p></o:p></p> <p class="MsoNormal">redstone.xmlrpc.XmlRpcFault: unhandled \
internal exception: could not execute query<o:p></o:p></p> <p \
class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)<o:p></o:p></p> \
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
javax.servlet.http.HttpServlet.service(HttpServlet.java:646)<o:p></o:p></p> <p \
class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)<o:p></o:p></p> <p \
class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)<o:p></o:p></p> \
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:100)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:57)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at \
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)<o:p></o:p></p> <p \
class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at \
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)<o:p></o:p></p>
 <p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at \
java.lang.Thread.run(Thread.java:745)<o:p></o:p></p> <p \
class="MsoNormal"><o:p>&nbsp;</o:p></p> <p class="MsoNormal">==&gt; \
/var/log/tomcat/catalina.2016-10-31.log &lt;==<o:p></o:p></p> <p \
class="MsoNormal">Oct 31, 2016 10:06:33 AM redstone.xmlrpc.XmlRpcDispatcher \
writeError<o:p></o:p></p> <p class="MsoNormal">WARNING: redstone.xmlrpc.XmlRpcFault: \
unhandled internal exception: could not execute query<o:p></o:p></p> <p \
class="MsoNormal"><o:p>&nbsp;</o:p></p> <p class="MsoNormal">I have tried turning up \
the verbosity in rhn.conf and restarting the services.&nbsp; Our DBA says it \
doesn&#8217;t look like the query is getting to the database.&nbsp; I get the same \
error through the website and spacecmd.<o:p></o:p></p> <p \
class="MsoNormal"><o:p>&nbsp;</o:p></p> <p class="MsoNormal">Any \
ideas?<o:p></o:p></p> <p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Thanks, Ian<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<hr>
<font face="Arial" size="2">This email is intended only for the addressee. It may \
contain confidential or proprietary information that cannot be disclosed without \
BCLC's permission. If you have received this email in error, please notify the sender \
immediately  and delete the email.</font>
</body>
</html>



_______________________________________________
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
--===============9194095179079188953==--


[prev in list] [next in list] [prev in thread] [next in thread] 

Configure | About | News | Add a list | Sponsored by KoreLogic