[prev in list] [next in list] [prev in thread] [next in thread]
List: esb-java-dev
Subject: Re: [Dev] [Architecture] Configuration for SHARD_ID column length in siddhi
From: Rukshan Premathunga <rukshan () wso2 ! com>
Date: 2019-11-25 8:22:02
Message-ID: CAAQqULUNZ1yEYskH0zwmXdVH760rpWF3iWYQcwEHvq2UovAQnQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Mon, Nov 25, 2019 at 1:08 PM Niveathika Rajendran <niveathika@wso2.com>
wrote:
> Hi Rukshan,
>
> As per the offline discussion with, @Suho, we have decided to ignore
> 'field.length' validation for SHARD_ID as it is much cleaner and this
> column is introduced by Siddhi Aggregation implementation. Please find the
> pull request here[1].
>
+1. Thanks for the quick solution.
>
> [1] https://github.com/siddhi-io/siddhi-store-rdbms/pull/214
>
> Best Regards,
> Niveathika
>
> On Mon, Nov 25, 2019 at 12:05 PM Niveathika Rajendran <niveathika@wso2.com>
> wrote:
>
>> Hi Rukshan and Fazlan,
>>
>> Regarding aggregation, since this config is based on the store type
>> (RDBMS), we cannot introduce this support in Siddhi Aggregation
>> implementation. However, we can introduce a system property config for
>> the siddhi-store-rdbms extension to ignore validation of 'field.length'
>> property. This will ensure that 'SHARD_ID' column can be added to the
>> 'field.length' property by default.
>>
>> @Fazlan, using the collation cannot be enabled by default, since we have
>> to maintain backward compatibility. However, this can be now enabled by a
>> single property '*use.collation'*.
>>
>> Best Regards,
>> Niveathika
>>
>>
>> On Sat, Nov 23, 2019 at 5:13 PM Fazlan Nazeem <fazlann@wso2.com> wrote:
>>
>>> One related and good to have feature is to set a single character
>>> encoding by default for the tables when siddhi creates them on the
>>> database. The primary key length issue is mostly due to having collations
>>> which take more bytes to represent a single character. If we can default to
>>> a character encoding of a single byte per character for each database type,
>>> then we could avoid these issues. Just a thought.
>>>
>>> On Fri, Nov 22, 2019 at 5:31 PM Rukshan Premathunga <rukshan@wso2.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> When we configure partitionById in SP, siddhi automatically adds
>>>> a SHARD_ID column to all the aggregated tables in RDBMS. But we are having
>>>> a "too long key issue" in the database. As a solution, we need to properly
>>>> set the column length for each attribute in the aggregated event stream.
>>>>
>>>> Limit of columns in the aggregated streams can be defined when we
>>>> implement the siddhi app. But SHARD_ID is used only when partitionById is
>>>> configured. So we cannot provide and initial column length for that.
>>>>
>>>> So it will be an ideal solution for us to have a configuration for
>>>> this. Otherwise, users need to alter the database or siddhi applications to
>>>> define these values. So can you please check the possibility to support
>>>> this?
>>>>
>>>> ex:
>>>> siddhi:
>>>> properties:
>>>> partitionById: true
>>>> shardId: dc1
>>>> shardId_size: 20 # or derive length from 'shardId'
>>>>
>>>> Thanks and Regards
>>>>
>>>> --
>>>> Rukshan C. Premathunga | Associate Technical Lead | WSO2 Inc.
>>>> (m) +94711822074 | (w) +94112145345 | Email: rukshan@wso2.com
>>>> GET INTEGRATION AGILE
>>>> Integration Agility for Digitally Driven Business
>>>> _______________________________________________
>>>> Dev mailing list
>>>> Dev@wso2.org
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>>
>>> *Fazlan Nazeem | *Associate Technical Lead | WSO2 Inc
>>> Mobile : +94772338839 | fazlann@wso2.com
>>>
>>>
>>> _______________________________________________
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>
>>
>> --
>> Niveathika Rajendran | Senior Software Engineer | WSO2 Inc.
>> (m) +94779037536 | (w) +9411743 5800 | (e) niveathika@wso2.com
>> <https://wso2.com/signature>
>>
>>
>
> --
> Niveathika Rajendran | Senior Software Engineer | WSO2 Inc.
> (m) +94779037536 | (w) +9411743 5800 | (e) niveathika@wso2.com
> <https://wso2.com/signature>
>
>
--
Rukshan C. Premathunga | Associate Technical Lead | WSO2 Inc.
(m) +94711822074 | (w) +94112145345 | Email: rukshan@wso2.com
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Nov 25, 2019 at 1:08 PM Niveathika Rajendran <<a \
href="mailto:niveathika@wso2.com">niveathika@wso2.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi \
Rukshan,<div><br></div><div>As per the offline discussion with, @Suho, we have \
decided to ignore 'field.length' validation for SHARD_ID as it is much \
cleaner and this column is introduced by Siddhi Aggregation implementation. Please \
find the pull request here[1].</div></div></blockquote><div>+1. Thanks for the quick \
solution. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div><br></div><div>[1] <a \
href="https://github.com/siddhi-io/siddhi-store-rdbms/pull/214" \
target="_blank">https://github.com/siddhi-io/siddhi-store-rdbms/pull/214</a></div><div><br></div><div>Best \
Regards,</div><div>Niveathika</div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Nov 25, 2019 at 12:05 PM Niveathika Rajendran <<a \
href="mailto:niveathika@wso2.com" target="_blank">niveathika@wso2.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi \
Rukshan and Fazlan,<div><br></div><div><div style="box-sizing:border-box"><font \
face="arial, sans-serif">Regarding aggregation, since this config is based on the \
store type (RDBMS), we cannot introduce this support in Siddhi Aggregation \
implementation. <span style="box-sizing:border-box;letter-spacing:0px">However, we \
can introduce a system property config for the siddhi-store-rdbms extension to \
ignore validation of 'field.length' property. </span><span \
style="box-sizing:border-box;letter-spacing:0px">This will ensure that \
'SHARD_ID' column can be added to the 'field.length' property by \
default.</span></font></div><div style="box-sizing:border-box"><div \
style="box-sizing:border-box"><font face="arial, \
sans-serif"><br></font></div></div><div style="box-sizing:border-box"><font \
face="arial, sans-serif">@Fazlan, using the collation cannot be enabled by default, \
since we have to maintain backward compatibility. However, this can be now enabled by \
a single property '<b>use.collation'</b>.</font></div><div \
style="box-sizing:border-box"><font face="arial, sans-serif"><br></font></div><div \
style="box-sizing:border-box"><font face="arial, sans-serif">Best Regards, \
</font></div><div style="box-sizing:border-box"><font face="arial, \
sans-serif">Niveathika</font></div><div style="box-sizing:border-box"><br \
style="color:rgb(28,30,41);font-family:"Akkurat \
Std",-apple-system,BlinkMacSystemFont,"Segoe \
UI",Roboto,Oxygen,Ubuntu,Cantarell,"Helvetica \
Neue",sans-serif;font-size:20.25px;white-space:pre-wrap"></div></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 23, 2019 at 5:13 PM \
Fazlan Nazeem <<a href="mailto:fazlann@wso2.com" \
target="_blank">fazlann@wso2.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr">One related and good to have \
feature is to set a single character encoding by default for the tables when siddhi \
creates them on the database. The primary key length issue is mostly due to having \
collations which take more bytes to represent a single character. If we can default \
to a character encoding of a single byte per character for each database type, then \
we could avoid these issues. Just a thought. </div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 22, 2019 at 5:31 PM \
Rukshan Premathunga <<a href="mailto:rukshan@wso2.com" \
target="_blank">rukshan@wso2.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div>When we \
configure partitionById in SP, siddhi automatically adds a SHARD_ID column to all \
the aggregated tables in RDBMS. But we are having a "too long key issue" in \
the database. As a solution, we need to properly set the column length for each \
attribute in the aggregated event stream. </div><div><br></div><div>Limit of \
columns in the aggregated streams can be defined when we implement the siddhi app. \
But SHARD_ID is used only when partitionById is configured. So we cannot provide \
and initial column length for that. </div><div><br></div><div>So it will be an ideal \
solution for us to have a configuration for this. Otherwise, users need to alter the \
database or siddhi applications to define these values. So can you please check the \
possibility to support this?</div><div><br></div><div>ex:</div><div>siddhi:<br> \
properties:<br> partitionById: true<br> shardId: dc1</div><div> \
shardId_size: 20 # or derive length from \
'shardId'</div><div><br></div><div>Thanks and Regards<br \
clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
style="color:rgb(136,136,136)"><div dir="ltr">Rukshan C. Premathunga | Associate \
Technical Lead | WSO2 Inc.<br></div></div><div dir="ltr" \
style="color:rgb(136,136,136)"><div>(m) +94711822074 | (w) +94112145345 | Email: <a \
href="mailto:rukshan@wso2.com" style="color:rgb(17,85,204)" \
target="_blank">rukshan@wso2.com</a></div><div><div><font color="#999999">GET \
INTEGRATION AGILE</font></div><div><font color="#999999">Integration Agility for \
Digitally Driven Business</font></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Dev mailing list<br>
<a href="mailto:Dev@wso2.org" target="_blank">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailman/listinfo/dev</a><br> \
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div \
dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div><span><font color="#888888">Thanks & \
Regards,</font></span></div><div><span><font \
color="#888888"><br></font></span></div><div dir="ltr"><span><font \
color="#888888"><b>Fazlan Nazeem | </b></font></span><span \
style="color:rgb(136,136,136)">Associate Technical Lead | </span><span \
style="color:rgb(136,136,136)">WSO2 Inc</span></div><div dir="ltr"><span><font \
color="#888888"><div>Mobile : +94772338839 | <a href="mailto:fazlann@wso2.com" \
target="_blank">fazlann@wso2.com</a></div><div><a><img \
src="https://c.content.wso2.com/signatures/wso2-signature-general.png" width="200" \
height="35"><br></a><br></div></font></span></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Architecture mailing list<br>
<a href="mailto:Architecture@wso2.org" target="_blank">Architecture@wso2.org</a><br>
<a href="https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture" \
rel="noreferrer" target="_blank">https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div \
dir="ltr"><div><div dir="ltr"><div><span></span><span></span>Niveathika Rajendran | \
Senior Software Engineer | WSO2 Inc.<br>(m) +94779037536 | (w) +9411743 5800 | (e) <a \
href="mailto:niveathika@wso2.com" \
target="_blank">niveathika@wso2.com</a></div><div><a \
href="https://wso2.com/signature" target="_blank"><img \
src="http://c.content.wso2.com/signatures/wso2-signature-general.png"></a><br></div><div><br></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div \
dir="ltr"><div><div dir="ltr"><div><span></span><span></span>Niveathika Rajendran | \
Senior Software Engineer | WSO2 Inc.<br>(m) +94779037536 | (w) +9411743 5800 | (e) <a \
href="mailto:niveathika@wso2.com" \
target="_blank">niveathika@wso2.com</a></div><div><a \
href="https://wso2.com/signature" target="_blank"><img \
src="http://c.content.wso2.com/signatures/wso2-signature-general.png"></a><br></div><div><br></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" \
class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div style="color:rgb(136,136,136)"><div \
dir="ltr">Rukshan C. Premathunga | Associate Technical Lead | WSO2 \
Inc.<br></div></div><div dir="ltr" style="color:rgb(136,136,136)"><div>(m) \
+94711822074 | (w) +94112145345 | Email: <a href="mailto:rukshan@wso2.com" \
style="color:rgb(17,85,204)" \
target="_blank">rukshan@wso2.com</a></div><div><div><font color="#999999">GET \
INTEGRATION AGILE</font></div><div><font color="#999999">Integration Agility for \
Digitally Driven Business</font></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic