[prev in list] [next in list] [prev in thread] [next in thread]
List: esb-java-dev
Subject: [Dev] Scaling APIM Analytics with data partitioning
From: Rukshan Premathunga <rukshan () wso2 ! com>
Date: 2020-01-02 10:33:12
Message-ID: CAAQqULX5=Adcpfw_OzyoXZ+w9QuQojmf1xJk5z+c_-8ps=VW1g () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (multipart/alternative)]
Hi All,
Due to the limitation of APIM analytics scaling, we were able to come up
with a solution based on the discussion we had in [1]. To scale APIM
analytics, which has configured alert, required following deployment.
Deployment is partitioning by analytics data, and we partitioned data by
tenant domain. So this solution works and can be scale when there are
multiple tenants in the deployment. But if it is a
single-tenant deployment and still need scaling, we can think of other
parameters like API name or provider to partition data.
[image: Untitled drawing.png]
GWs are configuring to publish events to receiver nodes. It is an
active-active deployment to support HA. Receiver nodes are
partitioned data, according to the tenant domain and forward to
the appropriate active-passive worker cluster.
To support this deployment, we need to apply configuration(siddhi app) to
receiver nodes. For each tenant(or partitioning group), there should be a
siddhi app where it filters the events by defined field name(tenant
domain). Also, the event sink configuration needs to define with its worker
cluster URLs.
The following is a sample configuration for a single tenant. Bold cloured
parameters need to change according to the tenant. So connection URLs and
tenant domains need to change in this configuration.
@App:name("APIM_EVENT_PUBLISHER_wso2_com")
@App:description("Receive the streams and passes it for inmemory sinks.")
-- org.wso2.apimgt.statistics.request:3.0.0
@sink(type='wso2event',
wso2.stream.id='org.wso2.apimgt.statistics.request:3.0.0',
*url="tcp://localhost:7613", auth.url="ssl://localhost:7713",*
protocol="thrift", username="admin", password="admin", mode="non-blocking"
, @map(type='wso2event'))
define stream OutputRequestStream (meta_clientType string,
applicationConsumerKey string, applicationName string, applicationId string,
applicationOwner string, apiContext string, apiName string, apiVersion
string, apiResourcePath string, apiResourceTemplate string,
apiMethod string, apiCreator string, apiCreatorTenantDomain string, apiTier
string, apiHostname string, username string, userTenantDomain string,
userIp string, userAgent string, requestTimestamp long, throttledOut bool,
responseTime long, serviceTime long, backendTime long, responseCacheHit
bool, responseSize long, protocol string, responseCode int, destination
string, securityLatency long, throttlingLatency long, requestMedLat long,
responseMedLat long, backendLatency long, otherLatency long, gatewayType
string, label string);
@source(type = 'inMemory' , topic = 'APIM_REQUEST')
define stream InComingRequestStream (meta_clientType string,
applicationConsumerKey string, applicationName string, applicationId string,
applicationOwner string, apiContext string, apiName string, apiVersion
string, apiResourcePath string, apiResourceTemplate string,
apiMethod string, apiCreator string, apiCreatorTenantDomain string, apiTier
string, apiHostname string, username string, userTenantDomain string,
userIp string, userAgent string, requestTimestamp long, throttledOut bool,
responseTime long, serviceTime long, backendTime long, responseCacheHit
bool, responseSize long, protocol string, responseCode int, destination
string, securityLatency long, throttlingLatency long, requestMedLat long,
responseMedLat long, backendLatency long, otherLatency long, gatewayType
string, label string);
-- org.wso2.apimgt.statistics.throttle:3.0.0
@sink(type='wso2event',
wso2.stream.id='org.wso2.apimgt.statistics.throttle:3.0.0',
*url="tcp://localhost:7613", auth.url="ssl://localhost:7713",*
protocol="thrift", username="admin", password="admin", mode="non-blocking"
, @map(type='wso2event'))
define stream OutputThrottledOutStream(meta_clientType string, username
string, userTenantDomain string, apiName string, apiVersion string,
apiContext string, apiCreator string, apiCreatorTenantDomain string,
applicationId string, applicationName string, subscriber string,
throttledOutReason string, gatewayType string, throttledOutTimestamp long,
hostname string);
@source(type = 'inMemory' , topic = 'ThrottledOutStream')
define stream InComingThrottledOutStream(meta_clientType string, username
string, userTenantDomain string, apiName string, apiVersion string,
apiContext string, apiCreator string, apiCreatorTenantDomain string,
applicationId string, applicationName string, subscriber string,
throttledOutReason string, gatewayType string, throttledOutTimestamp long,
hostname string);
-- org.wso2.apimgt.statistics.fault:3.0.0
@sink(type='wso2event',
wso2.stream.id='org.wso2.apimgt.statistics.fault:3.0.0',
*url="tcp://localhost:7613", auth.url="ssl://localhost:7713",*
protocol="thrift", username="admin", password="admin", mode="non-blocking"
, @map(type='wso2event'))
define stream OutputFaultStream(meta_clientType string,
applicationConsumerKey string, apiName string, apiVersion string,
apiContext string,
apiResourcePath string, apiMethod string, apiCreator string, username
string, userTenantDomain string, apiCreatorTenantDomain string,
hostname string, applicationId string, applicationName string, protocol
string, errorCode string, errorMessage string, requestTimestamp long);
@source(type = 'inMemory' , topic = 'FaultStream')
define stream IncomingFaultStream(meta_clientType string,
applicationConsumerKey string, apiName string, apiVersion string,
apiContext string,
apiResourcePath string, apiMethod string, apiCreator string, username
string, userTenantDomain string, apiCreatorTenantDomain string,
hostname string, applicationId string, applicationName string, protocol
string, errorCode string, errorMessage string, requestTimestamp long);
from InComingRequestStream*[apiCreatorTenantDomain == "wso2.com
<http://wso2.com>"]*
select *
insert into OutputRequestStream;
from InComingThrottledOutStream*[apiCreatorTenantDomain == "wso2.com
<http://wso2.com>"]*
select *
insert into OutputThrottledOutStream;
from IncomingFaultStream*[apiCreatorTenantDomain == "wso2.com
<http://wso2.com>"]*
select *
insert into OutputFaultStream;
Can you please review this solution and let us know if there are any
concerns.
[1] [API Analytics] Propose One Deployment Architecture That Works For The
Full Solution
Thanks and Regards
--
Rukshan C. Premathunga | Associate Technical Lead | WSO2 Inc.
(m) +94711822074 | (w) +94112145345 | Email: rukshan@wso2.com
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
[Attachment #7 (text/html)]
<div dir="ltr">Hi All,<div><br></div><div>Due to the limitation of APIM analytics \
scaling, we were able to come up with a solution based on the discussion we had in \
[1]. To scale APIM analytics, which has configured alert, required following \
deployment. </div><div>Deployment is partitioning by analytics data, and we \
partitioned data by tenant domain. So this solution works and can be scale when there \
are multiple tenants in the deployment. But if it is a single-tenant deployment and \
still need scaling, we can think of other parameters like API name or provider to \
partition data. </div><div><br></div><div><br></div><div><img src="cid:ii_k4wjycmh0" \
alt="Untitled drawing.png" width="542" height="407"><br></div><div><br></div><div>GWs \
are configuring to publish events to receiver nodes. It is an active-active \
deployment to support HA. Receiver nodes are partitioned data, according to the \
tenant domain and forward to the appropriate active-passive worker cluster. \
</div><div><br></div><div>To support this deployment, we need to apply \
configuration(siddhi app) to receiver nodes. For each tenant(or partitioning \
group), there should be a siddhi app where it filters the events by defined field \
name(tenant domain). Also, the event sink configuration needs to define with its \
worker cluster URLs.</div><div><br></div><div>The following is a sample configuration \
for a single tenant. Bold cloured parameters need to change according to the tenant. \
So connection URLs and tenant domains need to change in this \
configuration.</div><div><br></div><div><font color="#0000ff"><font \
face="monospace">@App:name("APIM_EVENT_PUBLISHER_wso2_com")</font><br><font \
face="monospace">@App:description("Receive the streams and passes it for \
</font>inmemory<font face="monospace"> sinks.")</font><br><br><font \
face="monospace">-- org.wso2.apimgt.statistics.request:3.0.0</font><br><font \
face="monospace">@sink(type='wso2event', <a \
href="http://wso2.stream.id">wso2.stream.id</a>='org.wso2.apimgt.statistics.request:3.0.0', \
</font><b style=""><font face="arial, \
sans-serif">url="tcp://localhost:7613", \
auth.url="ssl://localhost:7713",</font></b><br><font \
face="monospace">protocol="thrift", username="admin", \
password="admin", mode="non-blocking" , \
@map(type='wso2event'))</font><br><font face="monospace">define stream \
OutputRequestStream (meta_clientType string, applicationConsumerKey string, \
applicationName string, applicationId string,</font><br><font \
face="monospace">applicationOwner string, apiContext string, apiName string, \
apiVersion string, apiResourcePath string, apiResourceTemplate \
string,</font><br><font face="monospace">apiMethod string, apiCreator string, \
apiCreatorTenantDomain string, apiTier string, apiHostname string, username string, \
userTenantDomain string,</font><br><font face="monospace">userIp string, userAgent \
string, requestTimestamp long, throttledOut </font>bool<font face="monospace">, \
responseTime long, serviceTime long, backendTime long, \
responseCacheHit</font><br>bool<font face="monospace">, responseSize long, protocol \
string, responseCode int, destination string, securityLatency long, throttlingLatency \
long, requestMedLat long,</font><br><font face="monospace">responseMedLat long, \
backendLatency long, otherLatency long, gatewayType string, label \
string);</font><br><br><font face="monospace">@source(type = 'inMemory' , \
topic = 'APIM_REQUEST')</font><br><font face="monospace">define stream \
InComingRequestStream (meta_clientType string, applicationConsumerKey string, \
applicationName string, applicationId string,</font><br><font \
face="monospace">applicationOwner string, apiContext string, apiName string, \
apiVersion string, apiResourcePath string, apiResourceTemplate \
string,</font><br><font face="monospace">apiMethod string, apiCreator string, \
apiCreatorTenantDomain string, apiTier string, apiHostname string, username string, \
userTenantDomain string,</font><br><font face="monospace">userIp string, userAgent \
string, requestTimestamp long, throttledOut </font>bool<font face="monospace">, \
responseTime long, serviceTime long, backendTime long, \
responseCacheHit</font><br>bool<font face="monospace">, responseSize long, protocol \
string, responseCode int, destination string, securityLatency long, throttlingLatency \
long, requestMedLat long,</font><br><font face="monospace">responseMedLat long, \
backendLatency long, otherLatency long, gatewayType string, label \
string);</font><br><br><font face="monospace">-- \
org.wso2.apimgt.statistics.throttle:3.0.0</font><br><font \
face="monospace">@sink(type='wso2event', <a \
href="http://wso2.stream.id">wso2.stream.id</a>='org.wso2.apimgt.statistics.throttle:3.0.0', \
</font><b style=""><font face="arial, \
sans-serif">url="tcp://localhost:7613", \
auth.url="ssl://localhost:7713",</font></b><br><font \
face="monospace">protocol="thrift", username="admin", \
password="admin", mode="non-blocking" , \
@map(type='wso2event'))</font><br><font face="monospace">define stream \
OutputThrottledOutStream(meta_clientType string, username string, userTenantDomain \
string, apiName string, apiVersion string,</font><br><font \
face="monospace">apiContext string, apiCreator string, apiCreatorTenantDomain string, \
applicationId string, applicationName string, subscriber string,</font><br><font \
face="monospace">throttledOutReason string, gatewayType string, throttledOutTimestamp \
long, hostname string);</font><br><br><font face="monospace">@source(type = \
'inMemory' , topic = 'ThrottledOutStream')</font><br><font \
face="monospace">define stream InComingThrottledOutStream(meta_clientType string, \
username string, userTenantDomain string, apiName string, apiVersion \
string,</font><br><font face="monospace">apiContext string, apiCreator string, \
apiCreatorTenantDomain string, applicationId string, applicationName string, \
subscriber string,</font><br><font face="monospace">throttledOutReason string, \
gatewayType string, throttledOutTimestamp long, hostname string);</font><br><br><font \
face="monospace">-- org.wso2.apimgt.statistics.fault:3.0.0</font><br><font \
face="monospace">@sink(type='wso2event', <a \
href="http://wso2.stream.id">wso2.stream.id</a>='org.wso2.apimgt.statistics.fault:3.0.0', \
</font><b style=""><font face="arial, \
sans-serif">url="tcp://localhost:7613", \
auth.url="ssl://localhost:7713",</font></b><br><font \
face="monospace">protocol="thrift", username="admin", \
password="admin", mode="non-blocking" , \
@map(type='wso2event'))</font><br><font face="monospace">define stream \
OutputFaultStream(meta_clientType string, applicationConsumerKey string, apiName \
string, apiVersion string, apiContext string,</font><br><font \
face="monospace">apiResourcePath string, apiMethod string, apiCreator string, \
username string, userTenantDomain string, apiCreatorTenantDomain \
string,</font><br><font face="monospace">hostname string, applicationId string, \
applicationName string, protocol string, errorCode string, errorMessage string, \
requestTimestamp long);</font><br><br><font face="monospace">@source(type = \
'inMemory' , topic = 'FaultStream')</font><br><font \
face="monospace">define stream IncomingFaultStream(meta_clientType string, \
applicationConsumerKey string, apiName string, apiVersion string, apiContext \
string,</font><br><font face="monospace">apiResourcePath string, apiMethod string, \
apiCreator string, username string, userTenantDomain string, apiCreatorTenantDomain \
string,</font><br><font face="monospace">hostname string, applicationId string, \
applicationName string, protocol string, errorCode string, errorMessage string, \
requestTimestamp long);</font><br><br><font face="monospace">from \
InComingRequestStream</font><b style=""><font face="arial, \
sans-serif">[apiCreatorTenantDomain == "<a \
href="http://wso2.com">wso2.com</a>"]</font></b><br><font \
face="monospace">select *</font><br><font face="monospace">insert into \
OutputRequestStream;</font><br><br><font face="monospace">from \
InComingThrottledOutStream</font><b style=""><font face="arial, \
sans-serif">[apiCreatorTenantDomain == "<a \
href="http://wso2.com">wso2.com</a>"]</font></b><br><font \
face="monospace">select *</font><br><font face="monospace">insert into \
OutputThrottledOutStream;</font><br><br><font face="monospace">from \
IncomingFaultStream</font><b style=""><font face="arial, \
sans-serif">[apiCreatorTenantDomain == "<a \
href="http://wso2.com">wso2.com</a>"]</font></b><br><font \
face="monospace">select *</font><br><font face="monospace">insert into \
OutputFaultStream;</font></font><br></div><div><br></div><div>Can you please review \
this solution and let us know if there are any concerns.</div><div><br></div><div>[1] \
[API Analytics] Propose One Deployment Architecture That Works For The Full \
Solution</div><div><br></div><div>Thanks and Regards</div><div><br></div><div>-- \
<br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div \
dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div style="color:rgb(136,136,136)"><div dir="ltr">Rukshan C. Premathunga | \
Associate Technical Lead | WSO2 Inc.<br></div></div><div dir="ltr" \
style="color:rgb(136,136,136)"><div>(m) +94711822074 | (w) +94112145345 | Email: <a \
href="mailto:rukshan@wso2.com" style="color:rgb(17,85,204)" \
target="_blank">rukshan@wso2.com</a></div><div><div><font color="#999999">GET \
INTEGRATION AGILE</font></div><div><font color="#999999">Integration Agility for \
Digitally Driven Business</font></div></div></div></div></div></div></div></div></div></div></div></div></div>
--0000000000001b2518059b258ef6--
["Untitled drawing.png" (image/png)]
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic