[prev in list] [next in list] [prev in thread] [next in thread]
List: wsf-c-dev
Subject: Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10mill
From: Bhathiya Jayasekara <bhathiya () wso2 ! com>
Date: 2016-07-31 19:19:45
Message-ID: CACyz+v6EbzK1O8Kmhx4AjYnnv-sf13HjyAOH+OQ91B8z=YpD2w () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Ishara,
On Sun, Jul 31, 2016 at 1:04 PM, Ishara Karunarathna <isharak@wso2.com>
wrote:
>
> With 2s worm up and then 10s ramp up period
>
Earlier we had a ramp up period of 60s. Not sure that has any effect here.
Just mentioning.
Thanks,
Bhathiya
>>>>
>>>> Did you look at the behaviour of latency?
>>>>
>>>> Thanks
>>>>
>>>> Malith
>>>>
>>>>
>>>> On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
>>>> indunil@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We are currently engaged into a performance analysis where we are
>>>>> analyzing performance for User Add, Update, Authentication operations. The
>>>>> testing has been carried out in a following environment with 500
>>>>> concurrency and users up to 10 million.
>>>>>
>>>>> *Environment :*
>>>>>
>>>>> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
>>>>> MySQL 5.7
>>>>> Ubuntu 14.04
>>>>> Openldap-2.4.31
>>>>> IS 5.1.0
>>>>>
>>>>> In order to optimize the MYSQL server, following server parameters
>>>>> have been tuned accordingly. We have referred MYSQL documentation [1] as
>>>>> well as have performed analysis using several MYSQL tuners in [2].
>>>>>
>>>>> (1) *max_connections : 1000* (The maximum permitted number of
>>>>> simultaneous client connections.)
>>>>>
>>>>> (2) *join_buffer_size : 259968* (The minimum size of the buffer that
>>>>> is used for plain index scans, range index scans, and joins that do not use
>>>>> indexes and thus perform full table scans.)
>>>>>
>>>>> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
>>>>> memory area where InnoDB caches table and index data)
>>>>>
>>>>> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
>>>>> transactions that have not been committed yet)
>>>>>
>>>>> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
>>>>> instances. According to the mysql documentation[1], on systems with a large
>>>>> amount of memory, we can improve concurrency by dividing the buffer pool
>>>>> into multiple buffer pool instances. But couldn't change since it's a read
>>>>> only variable)
>>>>>
>>>>> (6) *key_buffer_size : 384000000* (size of the buffer used for index
>>>>> blocks)
>>>>>
>>>>> (7) *table_open_cache : 4000* (The number of open tables for all
>>>>> threads)
>>>>>
>>>>> (8) *sort_buffer_size : 4000000* (Each session that must perform a
>>>>> sort allocates a buffer of this size)
>>>>>
>>>>> (9) *read_buffer_size : 1000000* (Each thread that does a sequential
>>>>> scan for a table allocates a buffer of this size for each table it scans.
>>>>> If we do many sequential scans, we might want to increase this value)
>>>>>
>>>>> (10) *query_cache_type : 0 *
>>>>>
>>>>> (11) *query_cache_limit : 1048576* (Do not cache results that are
>>>>> larger than this number of bytes)
>>>>>
>>>>> (12) *query_cache_size : 1048576* (The amount of memory allocated for
>>>>> caching query results)
>>>>>
>>>>> (13) *thread_stack : 262144* (The stack size for each thread)
>>>>>
>>>>> (14) *net_buffer_length : 16384* (Each client thread is associated
>>>>> with a connection buffer and result buffer. Both begin with a size given by
>>>>> net_buffer_length but are dynamically enlarged up to max_allowed_packet
>>>>> bytes as needed)
>>>>>
>>>>> (15) *max_allowed_packet : 4194304* (The maximum size of one packet
>>>>> or any generated/intermediate string)
>>>>>
>>>>> (16) *thread_cache_size : 30* (no of threads the server should cache
>>>>> for reuse)
>>>>>
>>>>>
>>>>>
>>>>> IS has been configured as follows to optimize the performance.
>>>>>
>>>>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>>>>>
>>>>> *Xms : 2g *
>>>>>
>>>>> *Xmx : 2g *
>>>>>
>>>>> (2) Removed following entry from
>>>>> <IS_HOME>/repository/conf/tomcat/catalina-server.xml to disable http access
>>>>> logs.
>>>>>
>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>>> directory="${carbon.home}/repository/logs" prefix="http_access_"
>>>>> suffix=".log" pattern="combined" />
>>>>>
>>>>> (3) Tuned following parameters in axis2client.xml file.
>>>>>
>>>>> <parameter name="*defaultMaxConnPerHost*">1000</parameter>
>>>>>
>>>>> <parameter name="*maxTotalConnections*">30000</parameter>
>>>>>
>>>>> (4) Added following additional parameters to optimize database
>>>>> connection pool.
>>>>>
>>>>> <Property name="*maxWait*">60000</Property>
>>>>>
>>>>> <Property name="*maxActive*">600</Property>
>>>>>
>>>>> <Property name="*initialSize*">20</Property>
>>>>>
>>>>> (5) Tuning Tomcat parameters in
>>>>> <IS_HOME>/repository/conf/tomcat/catalina-server.xml.
>>>>>
>>>>> *acceptorThreadCount = 8 *
>>>>>
>>>>> *maxThreads="750" *
>>>>>
>>>>> *minSpareThreads="150" *
>>>>>
>>>>> *maxKeepAliveRequests="600" *
>>>>>
>>>>> *acceptCount="600"*
>>>>>
>>>>>
>>>>>
>>>>> JMeter has been configured as follows to optimize the performance.
>>>>>
>>>>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>>>>>
>>>>> *Xms : 1g *
>>>>>
>>>>> *Xmx : 1g *
>>>>>
>>>>>
>>>>> We were able to optimize the environment up to some level. But*
>>>>> currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s in
>>>>> around 6100000 user count.(User Add)*
>>>>>
>>>>> Appreciate your help on figuring out whether we need to do any
>>>>> modifications to the optimizations in MYSQL, IS and JMeter servers or to
>>>>> identify the exact issue for this sudden TPS dropping.
>>>>>
>>>>> [1] http://dev.mysql.com/doc/refman/5.7/en/optimizing-server.html
>>>>>
>>>>> [2] http://www.askapache.com/mysql/mysql-performance-tuning.html
>>>>>
>>>>>
>>>>> Thanks and Regards
>>>>> --
>>>>> Indunil Upeksha Rathnayake
>>>>> Software Engineer | WSO2 Inc
>>>>> Email indunil@wso2.com
>>>>> Mobile 0772182255
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Malith Jayasinghe
>>>>
>>>>
>>>> WSO2, Inc. (http://wso2.com)
>>>> Email : malithj@wso2.com
>>>> Mobile : 0770704040
>>>> Lean . Enterprise . Middleware
>>>>
>>>
>>>
>>>
>>> --
>>> Ishara Karunarathna
>>> Associate Technical Lead
>>> WSO2 Inc. - lean . enterprise . middleware | wso2.com
>>>
>>> email: isharak@wso2.com, blog: isharaaruna.blogspot.com, mobile:
>>> +94717996791
>>>
>>>
>>>
>>
>>
>> --
>> Malith Jayasinghe Ph.D.
>> Director
>>
>>
>> WSO2, Inc. (http://wso2.com)
>> Email : malithj@wso2.com
>> Mobile : 0770704040
>> Lean . Enterprise . Middleware
>>
>
>
>
> --
> Ishara Karunarathna
> Associate Technical Lead
> WSO2 Inc. - lean . enterprise . middleware | wso2.com
>
> email: isharak@wso2.com, blog: isharaaruna.blogspot.com, mobile:
> +94717996791
>
>
>
--
*Bhathiya Jayasekara*
*Senior Software Engineer,*
*WSO2 inc., http://wso2.com <http://wso2.com>*
*Phone: +94715478185*
*LinkedIn: http://www.linkedin.com/in/bhathiyaj
<http://www.linkedin.com/in/bhathiyaj>*
*Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>*
*Blog: http://movingaheadblog.blogspot.com
<http://movingaheadblog.blogspot.com/>*
[Attachment #5 (text/html)]
<div dir="ltr">Hi Ishara,<div class="gmail_extra"><br><div class="gmail_quote">On \
Sun, Jul 31, 2016 at 1:04 PM, Ishara Karunarathna <span dir="ltr"><<a \
href="mailto:isharak@wso2.com" target="_blank">isharak@wso2.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><div><br></div><div>With 2s worm up and then 10s ramp up period \
<br></div></div></div></div></blockquote><div><br></div><div>Earlier we had a ramp up \
period of 60s. Not sure that has any effect here. Just mentioning. \
</div><div><br></div><div>Thanks,</div><div>Bhathiya</div><div><br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><div></div><div><div class="h5"><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><div></div><div><div><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
dir="ltr"><div><br></div><div><br></div><div>Did you look at the behaviour of \
latency?</div><div><br></div><div>Thanks</div><div><br></div><div>Malith</div><br><div \
class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Jul 29, 2016 at 2:57 \
PM, Indunil Upeksha Rathnayake <span dir="ltr"><<a href="mailto:indunil@wso2.com" \
target="_blank">indunil@wso2.com</a>></span> \
wrote:<br></span><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div \
dir="ltr">Hi, <p style="margin-bottom:0in;line-height:100%">We are currently
engaged into a performance analysis where we are analyzing
performance for User Add, Update, Authentication operations. The
testing has been carried out in a following environment with 500
concurrency and users up to 10 million.
</p><p style="margin-bottom:0in;line-height:100%"><b>Environment
> </b><br></p><p style="margin-bottom:0in;line-height:100%">m3.2xlarge ( 8 core, \
> 30GB, SSD 2x80 GB) 3 instances.<br>MySQL
5.7<br>Ubuntu 14.04<br>Openldap-2.4.31<br>IS 5.1.0<br><br>
</p>
<p style="margin-bottom:0in;line-height:100%">In order to optimize
the MYSQL server, following server parameters have been tuned
accordingly. We have referred MYSQL documentation [1] as well as have
performed analysis using several MYSQL tuners in [2].
</p>
<p style="margin-bottom:0in;line-height:100%">(1) <b>max_connections
> 1000</b> (The maximum permitted number of simultaneous client
connections.)
</p>
<p style="margin-bottom:0in;line-height:100%">(2) <b>join_buffer_size
> 259968</b> (The minimum size of the buffer that is used for plain
index scans, range index scans, and joins that do not use indexes and
thus perform full table scans.)
</p>
<p style="margin-bottom:0in;line-height:100%">(3)
<b>innodb_buffer_pool_size : <a href="tel:5207959552" value="+15207959552" \
target="_blank">5207959552</a></b> (size of the memory area where InnoDB caches table \
and index data) </p>
<p style="margin-bottom:0in;line-height:100%">(4)
<b>innodb_log_buffer_size : 16777216</b> (size of the buffer for
transactions that have not been committed yet)
</p>
<p style="margin-bottom:0in;line-height:100%">(5)
<b>innodb_buffer_pool_instances : 1</b> (The number of buffer pool
instances. According to the mysql documentation[1], on systems with a
large amount of memory, we can improve concurrency by dividing the
buffer pool into multiple buffer pool instances. But couldn't change
since it's a read only variable)
</p>
<p style="margin-bottom:0in;line-height:100%">(6) <b>key_buffer_size
> 384000000</b> (size of the buffer used for index blocks)
</p>
<p style="margin-bottom:0in;line-height:100%">(7) <b>table_open_cache
> 4000</b> (The number of open tables for all threads)
</p>
<p style="margin-bottom:0in;line-height:100%">(8) <b>sort_buffer_size
> 4000000</b> (Each session that must perform a sort allocates a
buffer of this size)
</p>
<p style="margin-bottom:0in;line-height:100%">(9) <b>read_buffer_size
> 1000000</b> (Each thread that does a sequential scan for a table
allocates a buffer of this size for each table it scans. If we do
many sequential scans, we might want to increase this value)
</p>
<p style="margin-bottom:0in;line-height:100%">(10)
<b>query_cache_type : 0
</b></p>
<p style="margin-bottom:0in;line-height:100%">(11)
<b>query_cache_limit : 1048576</b> (Do not cache results that are
larger than this number of bytes)
</p>
<p style="margin-bottom:0in;line-height:100%">(12)
<b>query_cache_size : 1048576</b> (The amount of memory allocated for
caching query results)
</p>
<p style="margin-bottom:0in;line-height:100%">(13) <b>thread_stack
> 262144</b> (The stack size for each thread)
</p>
<p style="margin-bottom:0in;line-height:100%">(14)
<b>net_buffer_length : 16384</b> (Each client thread is associated
with a connection buffer and result buffer. Both begin with a size
given by net_buffer_length but are dynamically enlarged up to
max_allowed_packet bytes as needed)
</p>
<p style="margin-bottom:0in;line-height:100%">(15)
<b>max_allowed_packet : 4194304</b> (The maximum size of one packet
or any generated/intermediate string)
</p>
<p style="margin-bottom:0in;line-height:100%">(16)
<b>thread_cache_size : 30</b> (no of threads the server should cache
for reuse)
</p>
<p style="margin-bottom:0in;line-height:100%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%"><br></p><p \
style="margin-bottom:0in;line-height:100%">IS has been configured as follows to \
optimize the performance. </p>
<p style="margin-bottom:0in;line-height:100%">(1) JVM Heap
Settings (-Xms -Xmx) changed as follows:
</p>
<p style="margin-bottom:0in;line-height:100%"><b>Xms : 2g
</b></p>
<p style="margin-bottom:0in;line-height:100%"><b>Xmx : 2g
</b></p>
<p style="margin-bottom:0in;line-height:100%">
</p>
<p style="margin-bottom:0in;line-height:100%">(2) Removed
following entry from
<IS_HOME>/repository/conf/tomcat/catalina-server.xml to disable
http access logs.
</p>
<p style="margin-bottom:0in;line-height:100%"><Valve
className="org.apache.catalina.valves.AccessLogValve"
directory="${carbon.home}/repository/logs"
prefix="http_access_" suffix=".log"
pattern="combined" />
</p>
<p style="margin-bottom:0in;line-height:100%">
</p>
<p style="margin-bottom:0in;line-height:100%">(3) Tuned following
parameters in axis2client.xml file.
</p>
<p style="margin-bottom:0in;line-height:100%"><parameter
name="<b>defaultMaxConnPerHost</b>">1000</parameter>
</p>
<p style="margin-bottom:0in;line-height:100%"><parameter
name="<b>maxTotalConnections</b>">30000</parameter>
</p>
<p style="margin-bottom:0in;line-height:100%">
</p>
<p style="margin-bottom:0in;line-height:100%">(4) Added following
additional parameters to optimize database connection pool.
</p>
<p style="margin-bottom:0in;line-height:100%"><Property
name="<b>maxWait</b>">60000</Property>
</p>
<p style="margin-bottom:0in;line-height:100%"><Property
name="<b>maxActive</b>">600</Property>
</p>
<p style="margin-bottom:0in;line-height:100%"><Property
name="<b>initialSize</b>">20</Property>
</p>
<p style="margin-bottom:0in;line-height:100%">
</p>
<p style="margin-bottom:0in;line-height:100%">(5) Tuning Tomcat
parameters in <IS_HOME>/repository/conf/tomcat/catalina-server.xml.
</p>
<p style="margin-bottom:0in;line-height:100%"><b>acceptorThreadCount
= 8
</b></p>
<p style="margin-bottom:0in;line-height:100%"><b>maxThreads="750"
</b></p>
<p style="margin-bottom:0in;line-height:100%"><b>minSpareThreads="150"
</b></p>
<p style="margin-bottom:0in;line-height:100%"><b>maxKeepAliveRequests="600"
</b></p>
<p style="margin-bottom:0in;line-height:100%"><b>acceptCount="600"</b></p>
<br><br><br>JMeter has been
configured as follows to optimize the performance.
<p style="margin-bottom:0in;line-height:100%">(1) JVM Heap
Settings (-Xms -Xmx) changed as follows:
</p>
<p style="margin-bottom:0in;line-height:100%"><b>Xms : 1g
</b></p>
<p style="margin-bottom:0in;line-height:100%"><b>Xmx : 1g
</b></p>
<p style="margin-bottom:0in;line-height:100%">
</p>
<p style="margin-bottom:0in;line-height:100%"><br></p><p \
style="margin-bottom:0in;line-height:100%">We were able to optimize the environment \
up to some level. But<b> currently the TPS is dropping from the initial TPS 1139.5/s \
to 198.1/s in around 6100000 user count.(User Add)</b></p>
<p style="margin-bottom:0in;line-height:100%">Appreciate your help on figuring out \
whether we need to do any modifications to the optimizations in MYSQL, IS and JMeter
servers or to identify the exact issue for this sudden TPS dropping. <br></p>
<p style="margin-bottom:0in;line-height:100%">[1]
<a href="http://dev.mysql.com/doc/refman/5.7/en/optimizing-server.html" \
target="_blank">http://dev.mysql.com/doc/refman/5.7/en/optimizing-server.html</a> \
</p> <p style="margin-bottom:0in;line-height:100%">[2]
<a href="http://www.askapache.com/mysql/mysql-performance-tuning.html" \
target="_blank">http://www.askapache.com/mysql/mysql-performance-tuning.html</a></p> \
<p style="margin-bottom:0in;line-height:100%"><br></p><p \
style="margin-bottom:0in;line-height:100%">Thanks and Regards</p><span><font \
color="#888888">-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><font \
color="#888888"><div><span><font color="#888888"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div \
style="font-size:12.8px"><div><font color="#000000">Indunil Upeksha \
Rathnayake<br></font></div><div><span \
style="color:rgb(153,153,153);font-size:12.8px">Software Engineer | WSO2 \
Inc</span><br></div><div><span style="color:rgb(153,153,153);font-size:12.8px">Email \
<font color="#888888"><a href="mailto:indunil@wso2.com" \
target="_blank">indunil@wso2.com</a> <br></font></span></div><div><span \
style="color:rgb(153,153,153);font-size:12.8px"><font color="#888888">Mobile <a \
href="tel:0772182255" value="+94772182255" \
target="_blank">0772182255</a><br></font></span></div></div></div></div></div></div></ \
div></div></div></div></font></span></div></font></span></div></div></div></div></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div></div></div><span><font color="#888888"><br><br \
clear="all"><div><br></div>-- <br></font></span><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><span><font color="#888888">Malith Jayasinghe \
<br><br><br>WSO2, Inc. (<a href="http://wso2.com" \
target="_blank">http://wso2.com</a>)<br>Email : <a href="mailto:malithj@wso2.com" \
target="_blank">malithj@wso2.com</a><br>Mobile : <a href="tel:0770704040" \
value="+94770704040" target="_blank">0770704040</a> <br></font></span><span>Lean . \
Enterprise . Middleware</span></div></div></div></div></div></div> </div></div>
</blockquote></div></div></div><br><br clear="all"><span><br>-- <br><div \
data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><span></span><a href="http:///" \
target="_blank"></a><span></span>Ishara Karunarathna<br>Associate Technical \
Lead<br>WSO2 Inc. - lean . enterprise . middleware | <a href="http://wso2.com" \
target="_blank">wso2.com</a><br><br>email: <a href="mailto:isharak@wso2.com" \
target="_blank">isharak@wso2.com</a>, blog: <a \
href="http://isharaaruna.blogspot.com" target="_blank">isharaaruna.blogspot.com</a>, \
mobile: <a href="tel:%2B94717996791" value="+94717996791" \
target="_blank">+94717996791</a><br><br><img \
src="http://c.content.wso2.com/signatures/wso2-signature-general.png"><br></div></div></div></div></div></div></div></div>
</span></div></div>
</blockquote></div></div></div><span><font color="#888888"><br><br \
clear="all"><div><br></div>-- <br></font></span><div \
data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><span><font color="#888888">Malith Jayasinghe \
Ph.D.<br>Director</font></span><span><br><br><br>WSO2, Inc. (<a \
href="http://wso2.com" target="_blank">http://wso2.com</a>)<br>Email : <a \
href="mailto:malithj@wso2.com" target="_blank">malithj@wso2.com</a><br>Mobile : <a \
href="tel:0770704040" value="+94770704040" target="_blank">0770704040</a> <br>Lean . \
Enterprise . Middleware</span></div></div></div></div></div></div> </div></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><br>-- \
<br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span></span><a href="http:///" \
target="_blank"></a><span></span>Ishara Karunarathna<br>Associate Technical \
Lead<br>WSO2 Inc. - lean . enterprise . middleware | <a href="http://wso2.com" \
target="_blank">wso2.com</a><br><br>email: <a href="mailto:isharak@wso2.com" \
target="_blank">isharak@wso2.com</a>, blog: <a \
href="http://isharaaruna.blogspot.com" target="_blank">isharaaruna.blogspot.com</a>, \
mobile: <a href="tel:%2B94717996791" value="+94717996791" \
target="_blank">+94717996791</a><br><br><img \
src="http://c.content.wso2.com/signatures/wso2-signature-general.png"><br></div></div></div></div></div></div></div></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div \
class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div \
dir="ltr"><font face="courier new, monospace"><b><i>Bhathiya \
Jayasekara</i></b></font><div><font face="courier new, monospace"><i>Senior Software \
Engineer,</i></font></div><div><font face="courier new, monospace"><i>WSO2 inc., <a \
href="http://wso2.com" target="_blank">http://wso2.com</a></i></font></div><div><font \
face="courier new, monospace"><i><br></i></font></div><div><font face="courier new, \
monospace"><i>Phone: +94715478185</i></font></div><div><font face="courier new, \
monospace"><i>LinkedIn: <a href="http://www.linkedin.com/in/bhathiyaj" \
target="_blank">http://www.linkedin.com/in/bhathiyaj</a></i></font></div><div><font \
face="courier new, monospace"><i>Twitter: <a href="https://twitter.com/bhathiyax" \
target="_blank">https://twitter.com/bhathiyax</a></i></font></div><div><font \
face="courier new, monospace"><i>Blog: <a \
href="http://movingaheadblog.blogspot.com/" \
target="_blank">http://movingaheadblog.blogspot.com</a></i></font></div></div></div></div></div>
</div></div>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic