[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Delayed events processing / queue (anti-)pattern
From: Thunder Stumpges <thunder.stumpges () gmail ! com>
Date: 2015-03-27 14:55:47
Message-ID: CAB3xxCPXC1RyLTq+oXypDf=2Ag3dRjCTa83P35JYxHU4=npnmg () mail ! gmail ! com
[Download RAW message or body]
Yeah that's the one :) sorry, was on my phone and didn't want to look up
the exact name.
Cheers,
Thunder
On Mar 27, 2015 6:17 AM, "Brice Dutheil" <brice.dutheil@gmail.com> wrote:
> Would it help here to not actually issue a delete statement but instead
> use date based compaction and a dynamically calculated ttl that is some
> safe distance in the future from your key?
>
> I'm not sure about about this part *date based compaction*, do you mean
> DateTieredCompationStrategy ?
>
> Anyway we achieved something like that without this strategy with a TTL +
> date in partition key based approach. The thing however to watch is the
> size of the partition (one should avoid too long partitions (in thrift wide
> rows)), so care must be taken on the date increment to be correctly
> adjusted.
>
>
> -- Brice
>
> On Thu, Mar 26, 2015 at 5:23 PM, Robin Verlangen <robin@us2.nl> wrote:
>
>> Interesting thought, that should work indeed, I'll evaluate both options
>> and provide an update here once I have results.
>>
>> Best regards,
>>
>> Robin Verlangen
>> *Chief Data Architect*
>>
>> W http://www.robinverlangen.nl
>> E robin@us2.nl
>>
>> <http://goo.gl/Lt7BC>
>> *What is CloudPelican? <http://goo.gl/HkB3D>*
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>> On Thu, Mar 26, 2015 at 7:09 AM, Thunder Stumpges <
>> thunder.stumpges@gmail.com> wrote:
>>
>>> Would it help here to not actually issue a delete statement but instead
>>> use date based compaction and a dynamically calculated ttl that is some
>>> safe distance in the future from your key?
>>>
>>> Just a thought.
>>> -Thunder
>>> On Mar 25, 2015 11:07 AM, "Robert Coli" <rcoli@eventbrite.com> wrote:
>>>
>>>> On Wed, Mar 25, 2015 at 12:45 AM, Robin Verlangen <robin@us2.nl> wrote:
>>>>
>>>>> @Robert: can you elaborate a bit more on the "not ideal" parts? In my
>>>>> case I will be throwing away the rows (thus the points in time that are
>>>>> "now in the past"), which will create tombstones which are compacted away.
>>>>>
>>>>
>>>> "Not ideal" is what I mean... Cassandra has immutable data files, use
>>>> cases which do DELETE pay an obvious penalty. Some percentage of tombstones
>>>> will exist continuously, and you have to store them and seek past them.
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>
>
[Attachment #3 (text/html)]
<p dir="ltr">Yeah that's the one :) sorry, was on my phone and didn't want to \
look up the exact name. </p> <p dir="ltr">Cheers, <br>
Thunder <br>
</p>
<div class="gmail_quote">On Mar 27, 2015 6:17 AM, "Brice Dutheil" <<a \
href="mailto:brice.dutheil@gmail.com">brice.dutheil@gmail.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><blockquote \
style="margin:1.2em 0px;border-left-width:4px;border-left-style:solid;border-left-color:rgb(221,221,221);padding:0px \
1em;color:rgb(119,119,119);quotes:none"> <p style="margin:1.2em 0px!important">Would \
it help here to not actually issue a delete statement but instead use date based \
compaction and a dynamically calculated ttl that is some safe distance in the future \
from your key?</p> </blockquote>
<p style="margin:1.2em 0px!important">I'm not sure about about this part <em>date \
based compaction</em>, do you mean <code \
style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px \
0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid \
rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">DateTieredCompationStrategy</code> \
?</p> <p style="margin:1.2em 0px!important">Anyway we achieved something like that \
without this strategy with a TTL + date in partition key based approach. The thing \
however to watch is the size of the partition (one should avoid too long partitions \
(in thrift wide rows)), so care must be taken on the date increment to be correctly \
adjusted.</p> <div title="MDH:Jmd0O8KgPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTIuODAwMDAwMTkwNzM0OXB4OyI+V291bGQg
aXQgaGVscCBoZXJlIHRvIG5vdCBhY3R1YWxseSBpc3N1ZSBhIGRlbGV0ZSBzdGF0ZW1lbnQgYnV0
IGluc3RlYWQgdXNlIGRhdGUgYmFzZWQgY29tcGFjdGlvbiBhbmQgYSBkeW5hbWljYWxseSBjYWxj
dWxhdGVkIHR0bCB0aGF0IGlzIHNvbWUgc2FmZSBkaXN0YW5jZSBpbiB0aGUgZnV0dXJlIGZyb20g
eW91ciBrZXk/PC9zcGFuPjxkaXY+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTIuODAwMDAwMTkw
NzM0OXB4OyI+PGJyPjwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuIHN0eWxlPSJmb250LXNpemU6IDEy
LjgwMDAwMDE5MDczNDlweDsiPkknbSBub3Qgc3VyZSBhYm91dCBhYm91dCB0aGlzIHBhcnQgXzwv
c3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMi44MDAwMDAxOTA3MzQ5cHg7Ij5kYXRlIGJh
c2VkIGNvbXBhY3Rpb25fLCBkbyB5b3UgbWVhbiBgRGF0ZVRpZXJlZENvbXBhdGlvblN0cmF0ZWd5
YCA/PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTIuODAwMDAwMTkw
NzM0OXB4OyI+PGJyPjwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuIHN0eWxlPSJmb250LXNpemU6IDEy
LjgwMDAwMDE5MDczNDlweDsiPkFueXdheSB3ZSBhY2hpZXZlZCBzb21ldGhpbmcgbGlrZSB0aGF0
IHdpdGhvdXQgdGhpcyBzdHJhdGVneSB3aXRoIGEgVFRMICsgZGF0ZSBpbiBwYXJ0aXRpb24ga2V5
IGJhc2VkIGFwcHJvYWNoLiBUaGUgdGhpbmcgaG93ZXZlciB0byB3YXRjaCBpcyB0aGUgc2l6ZSBv
ZiB0aGUgcGFydGl0aW9uIChvbmUgc2hvdWxkIGF2b2lkIHRvbyBsb25nIHBhcnRpdGlvbnMgKGlu
IHRocmlmdCB3aWRlIHJvd3MpKSwgc28gY2FyZSBtdXN0IGJlIHRha2VuIG9uIHRoZSBkYXRlIGlu
Y3JlbWVudCB0byBiZSBjb3JyZWN0bHkgYWRqdXN0ZWQuPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4g
c3R5bGU9ImZvbnQtc2l6ZTogMTIuODAwMDAwMTkwNzM0OXB4OyI+PGJyPjwvc3Bhbj48L2Rpdj48
ZGl2PjxzcGFuIHN0eWxlPSJmb250LXNpemU6IDEyLjgwMDAwMDE5MDczNDlweDsiPjxicj48L3Nw
YW4+PC9kaXY+" style="min-height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div><div \
class="gmail_extra"><br clear="all"><div><div><div>-- Brice</div></div></div> \
<br><div class="gmail_quote">On Thu, Mar 26, 2015 at 5:23 PM, Robin Verlangen <span \
dir="ltr"><<a href="mailto:robin@us2.nl" \
target="_blank">robin@us2.nl</a>></span> wrote:<br><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
dir="ltr">Interesting thought, that should work indeed, I'll evaluate both \
options and provide an update here once I have results.</div><div \
class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr">Best regards, \
<br><div><br></div><div>Robin Verlangen</div><div><i>Chief Data \
Architect</i></div><div><i><br></i></div><div>W <a \
href="http://www.robinverlangen.nl" target="_blank"><font \
color="#666666">http://www.robinverlangen.nl</font></a></div><div>E <a \
href="mailto:robin@us2.nl" target="_blank"><font \
color="#666666">robin@us2.nl</font></a></div><div><br></div><div><a \
href="http://goo.gl/Lt7BC" target="_blank"><img \
src="http://static.cloudpelican.com/images/CloudPelican-email-signature.jpg"></a><br></div><div><i><a \
href="http://goo.gl/HkB3D" target="_blank"><font color="#cccccc">What is \
CloudPelican?</font></a></i></div><div><br></div><div><font \
color="#666666">Disclaimer: The information contained in this message and attachments \
is intended solely for the attention and use of the named addressee and may be \
confidential. If you are not the intended recipient, you are reminded that the \
information remains the property of the sender. You must not use, disclose, \
distribute, copy, print or rely on this e-mail. If you have received this message in \
error, please contact the sender immediately and irrevocably delete this message and \
any copies.</font></div></div></div></div> <br></span><div><div><div \
class="gmail_quote">On Thu, Mar 26, 2015 at 7:09 AM, Thunder Stumpges <span \
dir="ltr"><<a href="mailto:thunder.stumpges@gmail.com" \
target="_blank">thunder.stumpges@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Would it help here to not actually issue a \
delete statement but instead use date based compaction and a dynamically calculated \
ttl that is some safe distance in the future from your key? </p> <p dir="ltr">Just a \
thought. <br><span><font color="#888888">
-Thunder <br>
</font></span></p><div><div>
<div class="gmail_quote">On Mar 25, 2015 11:07 AM, "Robert Coli" <<a \
href="mailto:rcoli@eventbrite.com" target="_blank">rcoli@eventbrite.com</a>> \
wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div \
class="gmail_extra"><div class="gmail_quote">On Wed, Mar 25, 2015 at 12:45 AM, Robin \
Verlangen <span dir="ltr"><<a href="mailto:robin@us2.nl" \
target="_blank">robin@us2.nl</a>></span> wrote:<br><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
dir="ltr"><div>@Robert: can you elaborate a bit more on the "not ideal" \
parts? In my case I will be throwing away the rows (thus the points in time that are \
"now in the past"), which will create tombstones which are compacted \
away.</div></div></blockquote><div><br></div><div>"Not ideal" is what I \
mean... Cassandra has immutable data files, use cases which do DELETE pay an obvious \
penalty. Some percentage of tombstones will exist continuously, and you have to store \
them and seek past them.</div><div><br></div><div>=Rob</div><div> \
</div></div></div></div> </blockquote></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic