[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-dev
Subject:    Re: [DISCUSS] AMQP Large Messages
From:       Clebert Suconic <clebert.suconic () gmail ! com>
Date:       2017-11-16 20:52:37
Message-ID: CAKF+bsoSxrv6WsTFRbWXywL=fazjm17iPKxB-h3QZQHXvgZiYw () mail ! gmail ! com
[Download RAW message or body]


This is a bit more complex.


LargeServerMessafe is too tight with core.

Will be a bit challenging changing it. I will make large Message a property
of the message so it can be resumed in AMQP.

It will still be compatible change.  Won't need to change it to 3.0.

I will update the design here once I do some more work.

On Thu, Nov 16, 2017 at 9:46 AM Clebert Suconic <clebert.suconic@gmail.com>
wrote:

> I will try to make a conversion. I would rather treat than make another
> parameter.
>
> On Thu, Nov 16, 2017 at 9:31 AM Clebert Suconic <clebert.suconic@gmail.com>
> wrote:
>
>> There is no distinction.
>>
>> It will stream if the packet is more then min Large message size on
>> core.  Or max frame size in AMQP.
>>
>> I don't know how distinguish really large ones from the ones that are
>> just above the limit.
>>
>> On Thu, Nov 16, 2017 at 9:11 AM Martyn Taylor <mtaylor@redhat.com> wrote:
>>
>>> On Thu, Nov 16, 2017 at 2:01 PM, Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> wrote:
>>>
>>> > On Thu, Nov 16, 2017 at 6:10 AM, Martyn Taylor <mtaylor@redhat.com>
>>> wrote:
>>> > > Clebert,
>>> > >
>>> > > We need to distinguish between streamed messages and large
>>> messages.  For
>>> > > the basic large message case, i.e. a single large message sent to the
>>> > > broker.  I agree with what you have here.
>>> > >
>>> > > Streamed large messages (i.e. messages that are received in chunks)
>>> > allows
>>> > > us to store the message without having to keep the whole thing in
>>> broker
>>> > > memory.  This get's complicated with cross protocol as:
>>> > >
>>> > > 1. Not all protocols support streamed messages.
>>> > Exactly.. right now in Stomp.. we read the whole body in memory before
>>> > sending. Just like what is in master now for AMQP.
>>> >
>>> > > 2. Cross protocol may require message conversions
>>> > Just like in Stomp... we read the whole message in memory
>>> >
>>> > >
>>> > > Both 1. and 2. would likely mean that we'd likely need to read the
>>> whole
>>> > > message into memory and I'm not sure we really want to do this?  It
>>> > defeats
>>> > > one of the main purposes of streamed messages.  (The others being
>>> saving
>>> > > client memory and reducing the amount of data to resend on connection
>>> > drop.)
>>> >
>>> > Most times the user will have no control over this. Say you sent a
>>> > 100K + 1 byte over the configured limit, String messages in Core.  the
>>> > client will stream it as it being large
>>>
>>> This is why I think we need to distinguish between broker large messages
>>> and streamed messages.  This configuration option is really for streaming
>>> messages.  In our docs we talk about the ability to send messages that
>>> are
>>> larger than the broker memory using streaming.  What happens if I do this
>>> and try to consume it via STOMP or another protocol that doesn't support
>>> streaming?
>>>
>>> > .
>>> >
>>> >
>>> > >
>>> > > For 1. I wonder whether or not we want to support this at all, or if
>>> we
>>> > do,
>>> > > make it configurable.
>>> > > For 2. Is it possible to transform the message via a stream?  If not
>>> then
>>> > > perhaps we treat this as a raw bytes message?
>>> >
>>> > It is possible as I said.. I think it's a separate task. I may try to
>>> > do it within this scope.
>>> >
>>> OK sounds good.
>>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
>
-- 
Clebert Suconic


[prev in list] [next in list] [prev in thread] [next in thread]