[prev in list] [next in list] [prev in thread] [next in thread]
List: activemq-users
Subject: RE: Error enumerating large messages
From: John Lilley <john.lilley () redpointglobal ! com ! INVALID>
Date: 2023-07-31 13:08:01
Message-ID: CH3PR11MB72030917CC9CDC84A8B518BB9505A () CH3PR11MB7203 ! namprd11 ! prod ! outlook ! com
[Download RAW message or body]
Thanks!
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | \
john.lilley@redpointglobal.com<mailto:john.lilley@redpointglobal.com>
From: Clebert Suconic <clebert.suconic@gmail.com>
Sent: Friday, July 28, 2023 3:32 PM
To: users@activemq.apache.org
Subject: Re: Error enumerating large messages
*** [Caution] This email is from an external source. Please use caution responding, \
opening attachments or clicking embedded links. ***
I would upgrade to the latest version as I have done some substantial work with large \
messages and tests in 2.29.0: (e.g ARTEMIS-4233).
Besides this is a bit difficult to identify what you're client doing. The only thing \
I can think of is these fixes I had in 2.29. so I would get 2.30.
On Thu, Jul 27, 2023 at 8:38 PM John Lilley \
<john.lilley@redpointglobal.com.invalid<mailto:john.lilley@redpointglobal.com.invalid>> \
wrote: Greetings!
I have recently started enumerating messages in a queue from time to time, because \
we're trying to figure out if a unit of work is still pending. But I'm getting this \
error occasionally:
2023-07-27T21:38:30.007 [CanvasState-Abandoned] \
RpdmQueueUtils.browseStartMessages:159 [] ERROR - Error enumerating messages for pod \
{"message_type":"pod_specifier","pod_type":"project_execution","memory_mb":2048,"jvm_memory_mb":250} \
java.lang.RuntimeException: AMQ219023: The large message lost connection with its \
session, either because of a rollback or a closed session at \
org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(Clie \
ntLargeMessageImpl.java:91<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientL \
argeMessageImpl.java%3a91&c=E,1,EMAB08TWdYYO4LFWRLqdddz9Q_iAB7ChMRHrrEIJNdpVgnHLMM2sPA \
lVQqpMEtpzpR_YJMplv0cpDe_Ah4298DOeO--OpEZY9zjP9BOF4fWg3LGve2qthw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQBytesMessage.readBytes(ActiveMQByte \
sMessage.java:217<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQBytesMes \
sage.java%3a217&c=E,1,U8iiuF_0vvzFKT0OOuKIbN7FNmzqkVv5e8GTV98Lwki9ap5Ib0TJcYLCPgmyvpvwt1SI1hqGgxJ_dCfTCXyQSpi4uECf2uPKseho-csKsa0B&typo=1&ancr_add=1>)
at net.redpoint.rpdm.services.RpdmQueueUtils.browseStartMessages(RpdmQueueUtils.java: \
155<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRpdmQueueUtils.java%3a155&c=E, \
1,hFvYvhx-Bm7jX9I4bSkyUExXtlTOjbxHQ4ItkXy1yYQ5B7f9-WBh6CmX8Pv4pFUokof44Nji5MQQa6kvLxIGeJMGJHD-IrlgDvqxM6k7ukO3ShFOP34gW9caaw,,&typo=1&ancr_add=1>)
at net.redpoint.rpdm.services.RpdmQueueUtils.getAllQueuedCanvasIds(RpdmQueueUtils.jav \
a:76<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRpdmQueueUtils.java%3a76&c=E, \
1,PRn1WhZx6azSck3GU473g-9ktCSOR97NsDjrkiP3OJ5JQyMWj5C6UBEnwThoG_xk3DAN7ADM1uPtxdLVcvUr99q1B0UDxs4JYrId5pWmkKk3RDKT8y3rNuM,&typo=1&ancr_add=1>)
at net.redpoint.rpdm.canvas_state_query.CanvasStateQueryServerImpl.doCanvasAbandoned( \
CanvasStateQueryServerImpl.java:282<https://linkprotect.cudasvc.com/url?a=https%3a%2f% \
2fCanvasStateQueryServerImpl.java%3a282&c=E,1,0MQh_Hg6m_VB9wkxppYn2HByHmbQBz_0Rx2NC4eY \
dhXUhKEm6SWA6_8sTphQnaVqeYVGCacM3y7iQrP8nXwxHY-fQlzhvAs1uw1QuD81ixb_ryTjCDtK&typo=1&ancr_add=1>)
at net.redpoint.rpdm.canvas_state_query.CanvasStateQueryServerImpl$Abandoned.action(C \
anvasStateQueryServerImpl.java:265<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2 \
fCanvasStateQueryServerImpl.java%3a265&c=E,1,bvfVsjRdib5lJ43J88SQ29Sxp2Bpn3l1qL4J4SWoS \
aZ_O6Ctfc8E20zhvl_sVqGAsR1xPJyaebjiJcFq_ptrM_AFjm7-MPZtCnwwrns5_2bVkuYWVe37&typo=1&ancr_add=1>)
at net.redpoint.rpdm.services.LazyStartPeriodicThread.run<https://linkprotect.cudasvc \
.com/url?a=https%3a%2f%2fnet.redpoint.rpdm.services.LazyStartPeriodicThread.run&c=E,1, \
bnecENh1w9bBi6pzB-qxVHg9uPXowUJ6I9704_7OYZedkdrLHQ7EkvUS2umxkDmrw6PRsrEWgD7GpjjxAUnFaZ \
6_tiO9CdHO6OzVZ_6VfYDrcVmOETkv&typo=1&ancr_add=1>(LazyStartPeriodicThread.java:91<http \
s://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazyStartPeriodicThread.java%3a91&c=E, \
1,S_LcUAzeod145HEXSnkelQWJrrZecCtoxGDOL3xfr8mCp4-pAiPROYaP0iZMvvHC3RpC9cl3PQwadzu2NvYGyUKWSsXPy7XjofrmhXI-TS6rGQE,&typo=1&ancr_add=1>)
at java.base/java.lang.Thread.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2 \
fjava.lang.Thread.run&c=E,1,d3f8dPziDguMd4SJKV6eafc--V1uw-wct5EuGNVEoYVRHrz72swuyAKlJn \
5O_VoMIZ9Mo7sa4fnd-CzYay_khdKTWcC0WZRFpbcJXkLfug,,&typo=1&ancr_add=1>(Thread.java:833< \
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThread.java%3a833&c=E,1,Hxd07ytuTR \
hxKVOBsq2S5bA38NzJ62DEPJQF07nZYOnP-eg9kNSlLY4B1rOds_M5IQddK2jmK5fcmKjiK9sGna3cl3fz13CmqweNhY3-TWKoOoMJ0K0,&typo=1&ancr_add=1>)
Caused by: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219023: \
The large message lost connection with its session, either because of a rollback or a \
closed session] at org.apache.activemq.artemis.core.client.impl.LargeMessageController \
Impl.saveBuffer(LargeMessageControllerImpl.java:265<https://linkprotect.cudasvc.com/ur \
l?a=https%3a%2f%2fLargeMessageControllerImpl.java%3a265&c=E,1,1quz-1SHZC1IoSRTiHB03sEa \
l2tuBmA7SJl_0sa7wzLcGksj_4dxP2uEAvHk2pXuxJ9nnURmiODisln7UJbTbUwmj_DIsYwAObKyM94iKS1L3c00SPPHsDP45t4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(Cl \
ientLargeMessageImpl.java:157<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClie \
ntLargeMessageImpl.java%3a157&c=E,1,vq27LzS9s5yRES-dOKOFpVtbLWXc1JtCHQr2YAwe1IJNzTEvwt \
Cl3_8daFmvYG_2inn0BvXRzWxortesj76eFbMhrqlBD7bHWgI8d32ZYCDzu9u_5Q0,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer( \
ClientLargeMessageImpl.java:89<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCli \
entLargeMessageImpl.java%3a89&c=E,1,osSEY4cd7T7FrSSE-EhUE3aPXoV7wA6Tu7J36SGo19afHPoONk \
FPd5vQa21XDrjEjQVgt2Wv1ECI_OOoNsIF9d0dfwkgmUB3sPO4WvyG_zUs3Q,,&typo=1&ancr_add=1>)
This happens in two parts using JMS. The first enumerates the queue using \
browseMessages() to collect a List<Message>, and the second processes the Messages, \
reading their bodies and examining them. The problem occurs on this line in \
browseStartMessages when we read the body bytes bm.readBytes(requestBytes);
Despite the error text, the session isn't closed, or at least this error doesn't \
always happen despite my not doing anything to re-create the session. I suspect \
that, given the ClientLargeMessageImpl.getBodyBuffer on the stack, something is going \
wrong that is specific to "large" messages. IIRC the large messages use a coat-check \
pattern to store the message on disk and bypass the in-memory queue.
Can you suggest a solution? I've though of two potential changes:
- Raise minLargeMessageSize to something big enough that our messages are never \
"large"
- Read the body bytes immediately, while enumerating the queue, instead of later when \
looping over the List<Message>
I was also trying to see if there is a way to attach "metadata" to a message and only \
read that? These messages are large, and presumably it is expensive to scan them, \
and I'm really only looking for a UUID embedded in all of that.
JDK: Temurin 17
AMQ broker version: 2.28.0
AMQ JMS client dependency:
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>artemis-jms-client-all</artifactId>
<version>2.28.0</version>
</dependency>
Thanks
John
public List<Message> browseMessages(String queueName) {
// This helps avoid a known issue with queue message enumeration (in ActiveMQ \
"classic"), where after getting the final message // it hangs for a significant \
time, up to 20 seconds, before the Enumeration returns false. int count = \
getQueueEntryCount(queueName); List<Message> result = new ArrayList<>();
synchronized (lock) {
try {
Queue queue = getSession().createQueue(queueName);
try (QueueBrowser browser = getSession().createBrowser(queue)) {
Enumeration e = browser.getEnumeration();
while (count-- > 0 && e.hasMoreElements()) {
try {
result.add((Message) e.nextElement());
} catch (Exception ex) {
LOG.warn("Error browsing queue '" + queueName + "'", ex);
}
}
}
} catch (Exception ex) {
LOG.warn("Error browsing queue '" + queueName + "'", ex);
}
}
return result;
}
The second processes the Messages:
public List<RpcRequestPacket> browseStartMessages(PodSpecifier podSpecifier) {
List<RpcRequestPacket> result = new ArrayList<>();
for (var message : browseMessages(getStartQueueName(podSpecifier))) {
var bm = (BytesMessage)message;
try {
var requestBytes = new byte[(int) bm.getBodyLength()];
bm.readBytes(requestBytes);
result.add(new RpcRequestPacket(requestBytes, new \
JmsRpcMessageMetadata(message.getJMSMessageID(), ""))); }
catch (Exception ex) {
LOG.error("Error enumerating messages for pod " + podSpecifier, ex);
}
}
return result;
}
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
,1,dVmfqHI7KhMWjq-qYlMqDecRlRztXUKIY7-ob3A13WZmS4OA2MHx0VHKZyvbQBZC9TJd7QNsKJSxPBFM2w9iMo6FUIQwsQ_PM6uJg1w4pixLGB_5a6AVcL6-qmE,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | \
john.lilley@redpointglobal.com<mailto:john.lilley@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is confidential and \
is intended solely for the use of the individual(s) to whom it is addressed. If you \
believe you received this e-mail in error, please notify the sender immediately, \
delete the e-mail from your computer and do not copy, print or disclose it to anyone \
else. If you properly received this e-mail as a customer, partner or vendor of \
Redpoint, you should maintain its contents in confidence subject to the terms and \
conditions of your agreement(s) with Redpoint.
--
Clebert Suconic
PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is confidential and \
is intended solely for the use of the individual(s) to whom it is addressed. If you \
believe you received this e-mail in error, please notify the sender immediately, \
delete the e-mail from your computer and do not copy, print or disclose it to anyone \
else. If you properly received this e-mail as a customer, partner or vendor of \
Redpoint, you should maintain its contents in confidence subject to the terms and \
conditions of your agreement(s) with Redpoint.
[Attachment #3 (multipart/related)]
[Attachment #5 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"JetBrains Mono";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.gmailsignatureprefix
{mso-style-name:gmail_signature_prefix;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div class="sgensig"><br>
<br>
<table cellpadding="0" cellspacing="0" border="0" style="font-family: sans-serif; \
box-sizing: initial; max-width: 580px; color: #363636; border-collapse: collapse; \
line-height: 1.5;"> <tbody>
<tr>
<td valign="top">
<table width="500" cellpadding="0" cellspacing="0" border="0" style="font-family: \
sans-serif; border-collapse: collapse; color: #323232; width: 480px; line-height: \
16px;"> <tbody>
<tr>
<td valign="middle" width="95" style="padding-right: 10px;">
<p style="margin: 0.75pt; line-height: 0px;"><a \
href="https://www.redpointglobal.com/" target="_blank"><img \
src="cid:7c2d8d305acbc453d115748a1e32ab985d536270" height="67" width="67" \
style="border: 0; display: block; border-radius: 0px;" alt="rg"> </a></p>
</td>
<td valign="middle" width="600">
<table cellpadding="0" cellspacing="0" border="0" width="600" style="font-family: \
sans-serif; border-collapse: collapse; color: #323232; width: 600px; line-height: \
16px;"> <tbody>
<tr>
<td valign="top">
<p style="margin: 0.75pt; color: #a52418; font-size: 14px; font-family: Arial, \
sans-serif;font-weight:bold;"> John Lilley </p>
</td>
</tr>
<tr>
<td valign="top">
<p style="margin: 0.75pt; font-size: 13px; font-family: Arial, \
sans-serif;color:#000000;font-weight:bold;"> <span>Data Management Chief Architect, \
Redpoint Global Inc.</span> </p> </td>
</tr>
<tr>
<td valign="top">
<p style="margin: 0.75pt; font-size: 13px; font-family: Arial, \
sans-serif;color:#616161;"> <span>888 Worcester Street, Suite 200 Wellesley, MA \
02482</span> </p> </td>
</tr>
<tr>
<td valign="top">
<p style="margin: 0.75pt; font-size: 13px; font-family: Arial, \
sans-serif;color:#616161;"> <b>M: </b><a href="tel:+1 7209385761" \
style="text-decoration: none; color: #616161;">+1 7209385761</a> <span style="color: \
#a52418;">|</span> <a href="mailto:john.lilley@redpointglobal.com" \
style="text-decoration: underline; color: #2d62bb;"> \
john.lilley@redpointglobal.com</a> </p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Clebert Suconic <clebert.suconic@gmail.com> \
<br> <b>Sent:</b> Friday, July 28, 2023 3:32 PM<br>
<b>To:</b> users@activemq.apache.org<br>
<b>Subject:</b> Re: Error enumerating large messages<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:11.0pt;background:#EC8772">
<em><span style="font-family:"Arial",sans-serif;color:black">*** [Caution] \
This email is from an external source. Please use caution responding, opening \
attachments or clicking embedded links. ***</span></em><o:p></o:p></p> </div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
<div> <div>
<p class="MsoNormal">I would upgrade to the latest version as I have done some \
substantial work with large messages and tests in 2.29.0: (e.g ARTEMIS-4233). \
<o:p></o:p></p> <div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Besides this is a bit difficult to identify what you're client \
doing. The only thing I can think of is these fixes I had in 2.29. so I would get \
2.30.<o:p></o:p></p> </div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jul 27, 2023 at 8:38 PM John Lilley <<a \
href="mailto:john.lilley@redpointglobal.com.invalid">john.lilley@redpointglobal.com.invalid</a>> \
wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in"> <div>
<div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Greetings!<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have \
recently started enumerating messages in a queue from time to time, because we're \
trying to figure out if a unit of work is still pending. But I'm getting this \
error occasionally:<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2023-07-27T21:38:30.007 \
[CanvasState-Abandoned] RpdmQueueUtils.browseStartMessages:159 [] ERROR - Error \
enumerating messages for pod \
{"message_type":"pod_specifier","pod_type":"project_execution","memory_mb":2048,"jvm_memory_mb":250}
java.lang.RuntimeException: AMQ219023: The large message lost connection with its \
session, either because of a rollback or a closed session<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientLargeMessageImpl.java% \
3a91&c=E,1,EMAB08TWdYYO4LFWRLqdddz9Q_iAB7ChMRHrrEIJNdpVgnHLMM2sPAlVQqpMEtpzpR_YJMp \
lv0cpDe_Ah4298DOeO--OpEZY9zjP9BOF4fWg3LGve2qthw,,&typo=1&ancr_add=1">ClientLargeMessageImpl.java:91</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
org.apache.activemq.artemis.jms.client.ActiveMQBytesMessage.readBytes(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQBytesMessage.java%3a \
217&c=E,1,U8iiuF_0vvzFKT0OOuKIbN7FNmzqkVv5e8GTV98Lwki9ap5Ib0TJcYLCPgmyvpvwt1SI1hqG \
gxJ_dCfTCXyQSpi4uECf2uPKseho-csKsa0B&typo=1&ancr_add=1">ActiveMQBytesMessage.java:217</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
net.redpoint.rpdm.services.RpdmQueueUtils.browseStartMessages(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRpdmQueueUtils.java%3a155&am \
p;c=E,1,hFvYvhx-Bm7jX9I4bSkyUExXtlTOjbxHQ4ItkXy1yYQ5B7f9-WBh6CmX8Pv4pFUokof44Nji5MQQa6 \
kvLxIGeJMGJHD-IrlgDvqxM6k7ukO3ShFOP34gW9caaw,,&typo=1&ancr_add=1">RpdmQueueUtils.java:155</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
net.redpoint.rpdm.services.RpdmQueueUtils.getAllQueuedCanvasIds(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRpdmQueueUtils.java%3a76& \
;c=E,1,PRn1WhZx6azSck3GU473g-9ktCSOR97NsDjrkiP3OJ5JQyMWj5C6UBEnwThoG_xk3DAN7ADM1uPtxdL \
VcvUr99q1B0UDxs4JYrId5pWmkKk3RDKT8y3rNuM,&typo=1&ancr_add=1">RpdmQueueUtils.java:76</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
net.redpoint.rpdm.canvas_state_query.CanvasStateQueryServerImpl.doCanvasAbandoned(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCanvasStateQueryServerImpl.j \
ava%3a282&c=E,1,0MQh_Hg6m_VB9wkxppYn2HByHmbQBz_0Rx2NC4eYdhXUhKEm6SWA6_8sTphQnaVqeY \
VGCacM3y7iQrP8nXwxHY-fQlzhvAs1uw1QuD81ixb_ryTjCDtK&typo=1&ancr_add=1">CanvasStateQueryServerImpl.java:282</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
net.redpoint.rpdm.canvas_state_query.CanvasStateQueryServerImpl$Abandoned.action(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCanvasStateQueryServerImpl.j \
ava%3a265&c=E,1,bvfVsjRdib5lJ43J88SQ29Sxp2Bpn3l1qL4J4SWoSaZ_O6Ctfc8E20zhvl_sVqGAsR \
1xPJyaebjiJcFq_ptrM_AFjm7-MPZtCnwwrns5_2bVkuYWVe37&typo=1&ancr_add=1">CanvasStateQueryServerImpl.java:265</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at
<a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fnet.redpoint.rpdm.service \
s.LazyStartPeriodicThread.run&c=E,1,bnecENh1w9bBi6pzB-qxVHg9uPXowUJ6I9704_7OYZedkd \
rLHQ7EkvUS2umxkDmrw6PRsrEWgD7GpjjxAUnFaZ6_tiO9CdHO6OzVZ_6VfYDrcVmOETkv&typo=1&ancr_add=1">
net.redpoint.rpdm.services.LazyStartPeriodicThread.run</a>(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazyStartPeriodicThread.java \
%3a91&c=E,1,S_LcUAzeod145HEXSnkelQWJrrZecCtoxGDOL3xfr8mCp4-pAiPROYaP0iZMvvHC3RpC9c \
l3PQwadzu2NvYGyUKWSsXPy7XjofrmhXI-TS6rGQE,&typo=1&ancr_add=1">LazyStartPeriodicThread.java:91</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
java.base/<a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fjava.lang.Threa \
d.run&c=E,1,d3f8dPziDguMd4SJKV6eafc--V1uw-wct5EuGNVEoYVRHrz72swuyAKlJn5O_VoMIZ9Mo7 \
sa4fnd-CzYay_khdKTWcC0WZRFpbcJXkLfug,,&typo=1&ancr_add=1">java.lang.Thread.run</a>(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThread.java%3a833&c=E,1, \
Hxd07ytuTRhxKVOBsq2S5bA38NzJ62DEPJQF07nZYOnP-eg9kNSlLY4B1rOds_M5IQddK2jmK5fcmKjiK9sGna \
3cl3fz13CmqweNhY3-TWKoOoMJ0K0,&typo=1&ancr_add=1">Thread.java:833</a>)<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Caused by: \
ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219023: The large \
message lost connection with its session, either because of a rollback or a closed \
session]<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLargeMessageControllerImpl.j \
ava%3a265&c=E,1,1quz-1SHZC1IoSRTiHB03sEal2tuBmA7SJl_0sa7wzLcGksj_4dxP2uEAvHk2pXuxJ \
9nnURmiODisln7UJbTbUwmj_DIsYwAObKyM94iKS1L3c00SPPHsDP45t4,&typo=1&ancr_add=1">LargeMessageControllerImpl.java:265</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientLargeMessageImpl.java% \
3a157&c=E,1,vq27LzS9s5yRES-dOKOFpVtbLWXc1JtCHQr2YAwe1IJNzTEvwtCl3_8daFmvYG_2inn0Bv \
XRzWxortesj76eFbMhrqlBD7bHWgI8d32ZYCDzu9u_5Q0,&typo=1&ancr_add=1">ClientLargeMessageImpl.java:157</a>)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">at \
org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(<a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientLargeMessageImpl.java% \
3a89&c=E,1,osSEY4cd7T7FrSSE-EhUE3aPXoV7wA6Tu7J36SGo19afHPoONkFPd5vQa21XDrjEjQVgt2W \
v1ECI_OOoNsIF9d0dfwkgmUB3sPO4WvyG_zUs3Q,,&typo=1&ancr_add=1">ClientLargeMessageImpl.java:89</a>)<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This \
happens in two parts using JMS. The first enumerates the queue using \
browseMessages() to collect a List<Message>, and the second processes the \
Messages, reading their bodies and examining them. The problem occurs on this \
line in browseStartMessages when we read the body bytes<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
bm.readBytes(requestBytes);<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Despite \
the error text, the session isn't closed, or at least this error doesn't always \
happen despite my not doing anything to re-create the session. I suspect that, \
given the ClientLargeMessageImpl.getBodyBuffer on the stack, something is going \
wrong that is specific to "large" messages. IIRC the large messages use a \
coat-check pattern to store the message on disk and bypass the in-memory \
queue.<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can you \
suggest a solution? I've though of two potential changes:<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- Raise \
<em><span style="font-size:10.5pt;font-family:"Segoe \
UI",sans-serif;color:#1F2328;background:white">minLargeMessageSize</span></em> \
to something big enough that our messages are never "large"<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- Read \
the body bytes immediately, while enumerating the queue, instead of later when \
looping over the List<Message><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I was \
also trying to see if there is a way to attach "metadata" to a message and only read \
that? These messages are large, and presumably it is expensive to scan them, \
and I'm really only looking for a UUID embedded in all of that.<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">JDK: \
Temurin 17<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">AMQ \
broker version: 2.28.0<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">AMQ JMS \
client dependency:<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white"> <span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808"><</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">dependency</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">><br> \
\
<</span><span style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">groupId</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">>org.apache.activemq</</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">groupId</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">><br> \
\
<</span><span style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">artifactId</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">>artemis-jms-client-all</</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">artifactId</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">><br> \
\
<</span><span style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">version</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">>2.28.0</</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">version</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">><br> </</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#0033B3">dependency</span><span \
style="font-size:10.0pt;font-family:"JetBrains \
Mono";color:#080808">></span><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">John<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">public \
List<Message> browseMessages(String queueName) {<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
// This helps avoid a known issue with queue message enumeration (in ActiveMQ \
"classic"), where after getting the final message<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
// it hangs for a significant time, up to 20 seconds, before the Enumeration returns \
false.<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> int count = \
getQueueEntryCount(queueName);<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> List<Message> \
result = new ArrayList<>();<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> synchronized (lock) \
{<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> try \
{<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
Queue queue = getSession().createQueue(queueName);<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
try (QueueBrowser browser = getSession().createBrowser(queue)) {<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
Enumeration e = browser.getEnumeration();<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
while (count-- > 0 && e.hasMoreElements()) {<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
try {<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
result.add((Message) e.nextElement());<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
} catch (Exception ex) {<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
LOG.warn("Error browsing queue '" + queueName + "'", \
ex);<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
}<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
}<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
}<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> } catch \
(Exception ex) {<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
LOG.warn("Error browsing queue '" + queueName + "'", \
ex);<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
}<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> }<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
return result;<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">}<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The \
second processes the Messages:<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">public \
List<RpcRequestPacket> browseStartMessages(PodSpecifier podSpecifier) \
{<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
List<RpcRequestPacket> result = new ArrayList<>();<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
for (var message : browseMessages(getStartQueueName(podSpecifier))) {<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> var bm \
= (BytesMessage)message;<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> try \
{<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
var requestBytes = new byte[(int) bm.getBodyLength()];<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
bm.readBytes(requestBytes);<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
result.add(new RpcRequestPacket(requestBytes, new \
JmsRpcMessageMetadata(message.getJMSMessageID(), "")));<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
}<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> catch \
(Exception ex) {<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
LOG.error("Error enumerating messages for pod " + podSpecifier, \
ex);<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
}<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> }<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> \
return result;<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">}<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
</div> <p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" \
style="max-width:435.0pt;border-collapse:collapse;box-sizing:initial"> <tbody>
<tr>
<td valign="top" style="padding:0in 0in 0in 0in">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="480" \
style="width:5.0in;border-collapse:collapse"> <tbody>
<tr>
<td width="95" style="width:71.25pt;padding:0in 7.5pt 0in 0in">
<p style="margin:.75pt;mso-line-height-alt:0pt"><a \
href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&am \
p;c=E,1,dVmfqHI7KhMWjq-qYlMqDecRlRztXUKIY7-ob3A13WZmS4OA2MHx0VHKZyvbQBZC9TJd7QNsKJSxPBFM2w9iMo6FUIQwsQ_PM6uJg1w4pixLGB_5a6AVcL6-qmE,&typo=1" \
target="_blank"><span \
style="font-family:"Arial",sans-serif;text-decoration:none"><img border="0" \
width="67" height="67" style="width:.6979in;height:.6979in" id="_x0000_i1025" \
src="cid:1899e662cb278c893661" alt="rg"></span></a><span \
style="font-family:"Arial",sans-serif;color:#323232"><o:p></o:p></span></p> \
</td> <td width="600" style="width:6.25in;padding:0in 0in 0in 0in">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="600" \
style="width:6.25in;border-collapse:collapse"> <tbody>
<tr>
<td valign="top" style="padding:0in 0in 0in 0in">
<p style="margin:.75pt;line-height:12.0pt"><b><span \
style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#A52418">John \
Lilley <o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 0in 0in">
<p style="margin:.75pt;line-height:12.0pt"><b><span \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Data \
Management Chief Architect, Redpoint Global Inc. <o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 0in 0in">
<p style="margin:.75pt;line-height:12.0pt"><span \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#616161">888 \
Worcester Street, Suite 200 Wellesley, MA 02482 <o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0in 0in 0in 0in">
<p style="margin:.75pt;line-height:12.0pt"><b><span \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#616161">M: \
</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#616161"><a \
href="tel:+1%207209385761" target="_blank"><span \
style="color:#616161;text-decoration:none">+1 7209385761</span></a> </span><span \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#A52418">|</span><span \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#616161"> <a \
href="mailto:john.lilley@redpointglobal.com" target="_blank"><span \
style="color:#2D62BB">john.lilley@redpointglobal.com</span></a> \
<o:p></o:p></span></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><br>
PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is confidential and \
is intended solely for the use of the individual(s) to whom it is addressed. If you \
believe you received this e-mail in error, please notify the sender immediately, \
delete the e-mail from your computer and do not copy, print or disclose it to anyone \
else. If you properly received this e-mail as a customer, partner or vendor of \
Redpoint, you should maintain its contents in confidence subject to the terms and \
conditions of your agreement(s) with Redpoint. <o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span class="gmailsignatureprefix">-- </span><o:p></o:p></p>
<div>
<p class="MsoNormal">Clebert Suconic<o:p></o:p></p>
</div>
</div>
</div>
<!-- SGENORIGSIG --><br>
PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is confidential and \
is intended solely for the use of the individual(s) to whom it is addressed. If you \
believe you received this e-mail in error, please notify the sender immediately, \
delete the e-mail from your computer and do not copy, print or disclose it to anyone \
else. If you properly received this e-mail as a customer, partner or vendor of \
Redpoint, you should maintain its contents in confidence subject to the terms and \
conditions of your agreement(s) with Redpoint.
</body>
</html>
["logo.png" (image/png)]
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic