[prev in list] [next in list] [prev in thread] [next in thread]
List: calendarserver-changes
Subject: [CalendarServer-changes] [15439] CalendarServer/trunk/doc/Admin/LoadSimulation.rst
From: source_changes () macosforge ! org
Date: 2016-01-21 23:25:21
Message-ID: 20160121232522.60957140E86 () svn ! calendarserver ! org
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Revision: 15439
http://trac.calendarserver.org//changeset/15439
Author: dre@apple.com
Date: 2016-01-21 15:25:21 -0800 (Thu, 21 Jan 2016)
Log Message:
-----------
Update load simulation doc
Modified Paths:
--------------
CalendarServer/trunk/doc/Admin/LoadSimulation.rst
Modified: CalendarServer/trunk/doc/Admin/LoadSimulation.rst
===================================================================
--- CalendarServer/trunk/doc/Admin/LoadSimulation.rst 2016-01-14 22:00:31 UTC (rev \
15438)
+++ CalendarServer/trunk/doc/Admin/LoadSimulation.rst 2016-01-21 23:25:21 UTC (rev \
15439) @@ -28,19 +28,8 @@
./bin/sim
-The sim should begin logging its activity, which might look something like:
+The sim should begin logging its activity to stdout, a sample of which is shown in \
the next section.
-::
-
- Loaded 99 accounts.
- user01 - - - - - - - - - - - startup BEGIN
- user01 request 207✓[ 0.04 s] PROPFIND /principals/users/user01/
- user01 request 207✓[ 0.02 s] PROPFIND /principals/__uids__/user01/
- user01 request 200✓[ 0.01 s] REPORT /principals/
- user01 - - - - - - - - - - - poll BEGIN
- user01 request 207✓[ 0.19 s] PROPFIND /calendars/__uids__/user01/
- ...
-
- To stop the sim, type control-c.
---------------------
@@ -56,61 +45,119 @@
::
- user01 - - - - - - - - - - - startup BEGIN
- user01 request 207✓[ 0.04 s] PROPFIND /principals/users/user01/
- user01 request 207✓[ 0.02 s] PROPFIND /principals/__uids__/user01/
- user01 request 200✓[ 0.01 s] REPORT /principals/
+ Loaded 99 accounts.
+ user01 - - - - - - - - - - - startup: 10.11 BEGIN
+ user02 - - - - - - - - - - - startup: 10.11 BEGIN
+ user02 request 207✓[ 0.10 s] PROPFIND{principal} \
/principals/__uids__/10000000-0000-0000-0000-000000000002/ + user01 request 207✓[ \
0.11 s] PROPFIND{principal} \
/principals/__uids__/10000000-0000-0000-0000-000000000001/ + user02 request 200✓[ \
0.03 s] REPORT{pset} /principals/ + user02 - - - - - - - - - - - poll BEGIN
+ user01 request 200✓[ 0.03 s] REPORT{pset} /principals/
user01 - - - - - - - - - - - poll BEGIN
- user01 request 207✓[ 0.29 s] PROPFIND /calendars/__uids__/user01/
- user01 request 207✓[ 0.04 s] PROPFIND /calendars/__uids__/user01/calendar/
- user01 request 207✓[ 0.04 s] PROPFIND /calendars/__uids__/user01/inbox/
- user01 request 207✓[ 0.05 s] PROPFIND /calendars/__uids__/user01/tasks/
- user01 request 207✓[ 0.02 s] PROPFIND /calendars/__uids__/user01/outbox/
- user01 request 207✓[ 0.03 s] PROPFIND /calendars/__uids__/user01/dropbox/
- user01 request 207✓[ 0.03 s] PROPFIND /calendars/__uids__/user01/notification/
- user01 - - - - - - - - - - - poll END [ 0.51 s]
- user01 - - - - - - - - - - - startup END [ 0.59 s]
+ user02 request 207✓[ 0.20 s] PROPFIND{home} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/ + user01 request 207✓[ \
0.20 s] PROPFIND{home} /calendars/__uids__/10000000-0000-0000-0000-000000000001/ + \
user02 request 207✓[ 0.07 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/tasks/ + user01 request \
207✓[ 0.07 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/calendar/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/tasks/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/calendar/ + user02 request \
207✓[ 0.06 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/tasks/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/calendar/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/calendar/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/tasks/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/calendar/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/tasks/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/calendar/ + user02 request \
403✓[ 0.04 s] REPORT{sync-init} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/notification/ + user02 - - - \
- - - - - - - - poll END [ 0.59 s] + user02 - - - - - - - - - - - startup: 10.11 \
END [ 0.71 s] + user01 request 207✓[ 0.09 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/tasks/ + user01 request \
403✓[ 0.04 s] REPORT{sync-init} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/notification/ + user01 - - - \
- - - - - - - - poll END [ 0.63 s] + user01 - - - - - - - - - - - startup: 10.11 \
END [ 0.77 s]
-The first token on each line is the username associated with the activity. Most of \
the time, client activity is interleaved, so to trace a specific client you may wish \
to filter for a specific username. +The first token on each line is the username \
associated with the activity. Note that client activity is highly interleaved during \
normal operation.
-Client operations are groups of requests that are surrounded by lines with many \
dashes, either BEGIN or END. 'END' lines also include the overall duration of the \
operation, which is the elapsed wall clock time since the operation started. +Client \
operations are groups of requests that are delimited by lines with many dashes, \
either BEGIN or END. 'END' lines also include the overall duration of the operation, \
which is the elapsed wall clock time since the operation started.
Above, we see two behaviors that are nested; the 'startup' operation starts which \
includes a few requests and also triggers the 'poll' operation.
-Individual requests are logged with the word 'request', followed by the HTTP status, \
the request duration, the HTTP method of the request, and the URI that is being \
targeted. Successful requests are indicated by a ✓ while failed requests are \
indicated with —✗. +Individual requests are logged with the word 'request', \
followed by the HTTP status code, the request duration, the HTTP method of the \
request, some additional info about the request in curly braces (for example \
'principal' refers to the target of a PROPFIND, and sync-init is a type of REPORT), \
and the URI that is being targeted. Successful requests are indicated by a ✓ while \
failed requests are indicated with —✗.
Most client operations also log a 'lag' time when they start, e.g.
::
- user12 - - - - - - - - - - - create BEGIN {lag 0.80 ms}
+ user18 - - - - - - - - - - - invite BEGIN {lag 2.32 ms}
-'lag' in this context specifies the time elapsed between when the sim should have \
started this operation (based on internal timers / activity distributions) and when \
it actually did. This is intended to be a measure of client sim health; if the lag \
values begin to grow too high, that indicates that the client sim is 'slipping' and \
doesn't have enough CPU to perform its various activities at the rates specified in \
the config file. This is python, so a single instance of the client sim is limited to \
one CPU core. See the `Scalability`_ section for information about how to scale the \
load sim by combining multiple instances. +'lag' in this context specifies the time \
elapsed between when the sim should have started this operation (based on the sim \
configuration) and when it actually started. This is used as a measure of client sim \
health; if the lag values begin to grow too high, that indicates that the client sim \
is 'slipping' and is unable to perform its various activities at the rates specified \
in the config file. This is python 2.x, so a single instance of the client sim is \
limited to one CPU core. See the `Scalability`_ section for information about how to \
scale the load sim by combining multiple instances.
Summary Reports
---------------
When the sim stops or is killed, it emits a summary report that displays the total \
number of users that were active in this run, and overall counts for both individual \
requests and also the higher-level client operations. Also we display quality of \
service information used to grade the run as PASS or FAIL. An example report is shown \
below::
- Users : 20
- request count failed >3sec mean median
- DELETE 21 0 0 0.0186 0.0184
- GET 27 0 0 0.0341 0.0223
- POST 17 0 0 0.0709 0.0523
- PROPFIND 265 0 0 0.0593 0.0262
- PUT 44 0 2 0.3544 0.1735
- REPORT 107 0 0 0.0599 0.0280
+ ** REPORT **
+
+ * Client
+ Cpu Time : user 29.01 sys 4.34 total 00:00:33
+ Clients : 20
+ Qos : 0.7060
+ Run Time : 00:15:59
+ Start Time : 01/21 14:19:05
+ Users : 20
+
+ * Details
+ request count failed >0.1 sec >0.5 sec >1 sec \
>3 sec >5 sec >10 sec >30 sec mean median stddev QoS STATUS \
> + ---------------------------------------------------------------------------------- \
> ----------------------------------------------------------------------------------------
>
+ DELETE{event} 3 0 0 0 0 \
0 0 0 0 0.0715 0.0771 0.0101 0.3573 + GET \
3 0 2 0 0 0 0 0 \
0 0.1330 0.1381 0.0259 0.0000 + GET{event} \
219 0 117 1 0 0 0 0 \
0 0.1454 0.1056 0.0987 1.4541 + POST{attach} \
138 0 138 19 0 0 0 0 \
0 0.3243 0.2871 0.1502 0.0000 + POST{fb-small} \
1279 0 664 10 0 0 0 0 \
0 0.1299 0.1026 0.0807 0.2598 + POST{share-calendar} \
60 0 60 5 0 0 0 0 \
0 0.2990 0.2686 0.1206 0.0000 + PROPFIND{home} \
20 0 20 0 0 0 0 0 \
0 0.1534 0.1454 0.0204 0.6136 + PROPFIND{principal} \
20 0 1 0 0 0 0 0 \
0 0.0798 0.0829 0.0125 0.9319 + PROPPATCH{calendar} \
120 0 2 0 0 0 0 0 \
0 0.0608 0.0569 0.0124 0.6216 + PUT{attendee-medium} \
2 0 2 0 0 0 0 0 \
0 0.1585 0.1628 0.0044 0.2426 + PUT{attendee-small} \
1 0 1 1 0 0 0 0 \
0 0.6156 0.6156 0.0000 1.2312 + PUT{event} \
220 0 220 40 10 0 0 0 \
0 0.3760 0.2587 0.3363 1.5041 + PUT{organizer-large} \
7 0 6 1 0 0 0 0 \
0 0.2066 0.1865 0.1353 0.4050 + PUT{organizer-medium} \
111 0 81 8 2 0 0 0 \
0 0.2419 0.1381 0.2725 0.5354 + PUT{organizer-small} \
22 0 13 1 0 0 0 0 \
0 0.1792 0.1091 0.1762 0.7168 + PUT{update} \
78 0 78 6 0 0 0 0 \
0 0.3087 0.2923 0.1048 1.2346 + REPORT{cpsearch} \
1279 0 63 0 0 0 0 0 \
0 0.0482 0.0378 0.0385 0.0000 + REPORT{multiget-small} \
15 0 0 0 0 0 0 0 \
0 0.0510 0.0496 0.0052 0.2039 + REPORT{pset} \
20 0 0 0 0 0 0 0 \
0 0.0300 0.0302 0.0008 0.9120 + REPORT{sync-init} \
20 0 0 0 0 0 0 0 \
0 0.0427 0.0394 0.0099 0.5854 + REPORT{sync} \
15 0 0 0 0 0 0 0 \
0 0.0481 0.0471 0.0032 0.1925 +
+ operation count failed >0.1 sec >0.5 sec >1 sec \
>3 sec >5 sec >10 sec >30 sec mean median stddev avglag (ms) \
> STATUS
+ ------------------------------------------------------------------------------------ \
------------------------------------------------------------------------------------------
+ accept 3 0 3 1 0 \
0 0 0 0 0.4447 0.2947 0.2365 7.8805 + \
create 220 0 220 40 10 \
0 0 0 0 0.3780 0.2615 0.3363 13.1159 + \
download 3 0 3 0 0 \
0 0 0 0 0.1515 0.1647 0.0343 0.0000 + \
invite 140 0 140 136 105 \
22 3 0 0 1.8759 1.5909 1.1846 2.3190 \
+ poll 20 20 20 20 0 \
0 0 0 0 0.6612 0.6573 0.0878 0.0000 FAIL + \
push 447 0 0 0 0 \
0 0 0 0 0.0002 0.0001 0.0004 0.0000 + \
startup: 10.11 20 0 20 20 0 \
0 0 0 0 0.7734 0.7687 0.0903 0.0000 + \
update{description} 38 0 38 19 0 \
0 0 0 0 0.5170 0.5315 0.1834 41.3110 + \
update{title} 40 0 40 17 1 \
0 0 0 0 0.5271 0.4719 0.1920 28.1850 + \
+ *** FAIL
+ Greater than 1% POLL failed
+ Exit code: 1
+
- operation count failed >3sec mean median avglag (ms)
- accept 10 0 0 0.2808 0.2942 0.8490
- create 17 0 0 0.0560 0.0484 1.0024
- invite 17 0 2 0.8713 0.4774 0.9585
- poll 64 0 0 0.3856 0.3234 0.0000
- reply done 12 0 0 0.0177 0.0174 1.9181
- startup 20 0 0 0.7369 0.6023 0.0000
- PASS
-The pass / fail criteria are defined in `contrib/performance/loadtest/profiles.py \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/profiles.py>`_ \
for operations and `contrib/performance/loadtest/population.py \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/population.py>`_ \
for individual requests, and are generally derived from execution time and failure \
rate. +The pass / fail criteria are defined in \
`contrib/performance/loadtest/thresholds.json \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/thresholds.json>`_. \
This json data describes the maximum percentage ("thresholds") of each request and \
operation type that are allowed in each time bucket ("limits"), which if exceeded \
will cause that type to be failed. For example, the configuration for requests uses \
the following buckets, which correspond to the time buckets in the report: (values in \
seconds)
+``[ 0.1, 0.5, 1.0, 3.0, 5.0, 10.0, 30.0]``
+
+The PUT{event} threshold configuration states:
+
+``[ 100.0, 100.0, 100.0, 75.0, 50.0, 25.0, 0.5]``
+
+This means the PUT{event} type is considered too slow if more than 75% of them take \
longer than 3 seconds, or more than 50% take longer than 5 seconds, or more than 25% \
take longer than 10 seconds, or more than .5% take longer than 30 seconds. Setting a \
bucket to 100% effectively ignores that bucket in calculating the pass / fail \
judgement. If it seems like these values might be somewhat arbitrary, that's because \
they are. +
---------------------
Configuration
---------------------
@@ -129,12 +176,39 @@
Server Specification
---------------------
-Specify the URI to which the client sim should connect, e.g::
+The client sim supports testing of 'podded' environments. If you aren't using pods, \
not to worry, the default configuration still works. For each pod, define the uri and \
other server-specific attributes such as the amp push host and port (used for client \
notifications in lieu of APNS), and the stats socket port (if enabled in the server \
config).
- <!-- Identify the server to be load tested. -->
- <key>server</key>
- <string>https://127.0.0.1:8443/</string>
+::
+ <key>servers</key>
+ <dict>
+ <key>PodA</key>
+ <dict>
+ <key>enabled</key>
+ <true/>
+
+ <!-- Identify the server to be load tested. -->
+ <key>uri</key>
+ <string>https://localhost:8443</string>
+
+ <key>ampPushHosts</key>
+ <array>
+ <string>localhost</string>
+ </array>
+ <key>ampPushPort</key>
+ <integer>62311</integer>
+
+ <!-- Define whether server supports stats socket. -->
+ <key>stats</key>
+ <dict>
+ <key>enabled</key>
+ <true/>
+ <key>Port</key>
+ <integer>8100</integer>
+ </dict>
+ </dict>
+ ...
+
User Accounts
-------------
@@ -142,148 +216,96 @@
::
- <!-- Define the credentials of the clients which will be used to \
load test
- the server. These credentials must already be valid on the \
server. -->
- <key>accounts</key>
- <dict>
- <!-- The loader is the fully-qualified Python name of a \
callable which
- returns a list of directory service records defining \
all of the client accounts
- to use. \
contrib.performance.loadtest.sim.recordsFromCSVFile reads username,
- password, mailto triples from a CSV file and returns \
them as a list of faked
- directory service records. -->
- <key>loader</key>
- \
<string>contrib.performance.loadtest.sim.recordsFromCSVFile</string> + \
<key>accounts</key> + <dict>
+ <!-- The loader is the fully-qualified Python name of a callable which
+ returns a list of directory service records defining all of the \
client accounts + to use. \
contrib.performance.loadtest.sim.recordsFromCSVFile reads username, + \
password, mailto triples from a CSV file and returns them as a list of faked + \
directory service records. --> + <key>loader</key>
+ <string>contrib.performance.loadtest.sim.recordsFromCSVFile</string>
- <!-- Keyword arguments may be passed to the loader. -->
- <key>params</key>
- <dict>
- <!-- recordsFromCSVFile interprets the path relative \
to the config.plist,
- to make it independent of the script's \
working directory while still allowing
- a relative path. This isn't a great \
solution. -->
- <key>path</key>
- \
<string>contrib/performance/loadtest/accounts.csv</string>
- </dict>
- </dict>
+ <!-- Keyword arguments may be passed to the loader. -->
+ <key>params</key>
+ <dict>
+ <!-- recordsFromCSVFile interprets the path relative to the \
config.plist, + to make it independent of the script's working \
directory while still allowing + a relative path. This isn't a \
great solution. --> + <key>path</key>
+ <string>contrib/performance/loadtest/accounts.csv</string>
+ <!-- When there are accounts for multiple pods, interleave the \
accounts for each + pod so that the arrival mechanism will cycle \
clients between each pod. --> + <key>interleavePods</key>
+ <true/>
+ </dict>
+ </dict>
+
+
The accounts.csv file has lines like shown below::
- user01,user01,User 01,user01@example.com
+ user01,user01,User 01,user01@example.com,10000000-0000-0000-0000-000000000001,PodA
Client Arrival
----------------
This section configures the number of accounts to use, and defines how quickly \
clients are initialized when the sim starts::
- <!-- Define how many clients will participate in the load test and \
how
- they will show up. -->
- <key>arrival</key>
- <dict>
+ <!-- Define how many clients will participate in the load test and how
+ they will show up. -->
+ <key>arrival</key>
+ <dict>
- <!-- Specify a class which creates new clients and \
introduces them into
- the test. \
contrib.performance.loadtest.population.SmoothRampUp introduces
- groups of new clients at fixed intervals up to a \
maximum. The size of the
- group, interval, and maximum are configured by the \
parameters below. The
- total number of clients is groups * groupSize, which \
needs to be no larger
- than the number of credentials created in the \
accounts section. -->
- <key>factory</key>
- \
<string>contrib.performance.loadtest.population.SmoothRampUp</string> + <!-- \
Specify a class which creates new clients and introduces them into + the \
test. contrib.performance.loadtest.population.SmoothRampUp introduces + \
groups of new clients at fixed intervals up to a maximum. The size of the + \
group, interval, and maximum are configured by the parameters below. The + \
total number of clients is groups * groupSize, which needs to be no larger + \
than the number of credentials created in the accounts section. --> + \
<key>factory</key> + \
<string>contrib.performance.loadtest.population.SmoothRampUp</string>
- <key>params</key>
- <dict>
- <!-- groups gives the total number of groups of \
clients to introduce. -->
- <key>groups</key>
- <integer>20</integer>
+ <key>params</key>
+ <dict>
+ <!-- groups gives the total number of groups of clients to introduce. -->
+ <key>groups</key>
+ <integer>10</integer>
- <!-- groupSize is the number of clients in each \
group of clients. It's
- really only a "smooth" ramp up if this is \
pretty small. -->
- <key>groupSize</key>
- <integer>1</integer>
+ <!-- groupSize is the number of clients in each group of clients. It's
+ really only a "smooth" ramp up if this is pretty small. -->
+ <key>groupSize</key>
+ <integer>2</integer>
- <!-- Number of seconds between the introduction of \
each group. -->
- <key>interval</key>
- <integer>3</integer>
- </dict>
+ <!-- Number of seconds between the introduction of each group. -->
+ <key>interval</key>
+ <integer>3</integer>
- </dict>
+ <!-- Number of clients each user is assigned to. -->
+ <!-- Set weight of clients to 1 if this is > 1. Number of clients must \
match this value if > 1. --> + <key>clientsPerUser</key>
+ <integer>1</integer>
+ </dict>
-In the default configuration, one client is initialized every 3 seconds, until 20 \
clients are initialized. As soon as a client is initialized, it begins to perform its \
specified behaviors at the configured rates (see "Client Behaviors"). + </dict>
-To increase the client load, increase the number of groups.
-To increase the rate at which clients are initialized, reduce 'interval' and / or \
increase 'groupSize' +In the default configuration, three clients are initialized \
every 3 seconds, until 30 clients are initialized (groups * groupSize). As soon as a \
client is initialized, it begins to perform its specified behaviors at the configured \
rates (see "Client Behaviors").
-Remember: **The total number of clients is groups * groupSize, which needs to be no \
larger than the number of credentials created in the accounts section** +To increase \
the client load, increase the number of groups and / or groupSize. Take care not to \
exceed the number of accounts defined in accounts.csv.
+To increase the rate at which clients are initialized, reduce 'interval'.
+
Client Behaviors
----------------
-The 'clients' plist key is an array of dictionaries, where each dict has the keys:
+Client behaviors are defined in `contrib/performance/loadtest/clients.plist \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/clients.plist>`_. \
The 'clients' plist key is an array of dictionaries describing the client. The \
clients.plist is well commented, so no need to repeat those details here.
-- 'software', which specifies the implementation class for this client. For example:
+- 'profiles' is an array of dictionaries specifying individual behaviors of each \
client. Each dict has a 'class' key which specifies the implementation class for this \
behavior, and a 'params' dict with options specific to that behavior. See the plist \
for more information.
-::
-
- <key>software</key>
- <string>contrib.performance.loadtest.ical.SnowLeopard</string>
-
-- 'params', optionally specifying parameters accepted by this software. For example:
-
-::
-
- <!-- Arguments to use to initialize the SnowLeopard instance. -->
- <key>params</key>
- <dict>
- <!-- SnowLeopard can poll the calendar home at some interval. This is
- in seconds. -->
- <key>calendarHomePollInterval</key>
- <integer>30</integer>
-
- <!-- If the server advertises xmpp push, SnowLeopard can wait for \
notifications
- about calendar home changes instead of polling for them \
periodically. If
- this option is true, then look for the server advertisement for \
xmpp push
- and use it if possible. Still fall back to polling if there is no \
xmpp push
- advertised. -->
- <key>supportPush</key>
- <false />
- </dict>
-
-- 'profiles' is an array of dictionaries specifying individual behaviors of this \
software. Each dict has a 'class' key which specifies the implementation class for \
this behavior, and a 'params' dict with options specific to that behavior. For \
example:
-
-::
-
- <!-- This profile accepts invitations to events, handles cancels, and
- handles replies received. -->
- <dict>
- <key>class</key>
- <string>contrib.performance.loadtest.profiles.Accepter</string>
-
- <key>params</key>
- <dict>
- <key>enabled</key>
- <true/>
-
- <!-- Define how long to wait after seeing a new invitation before
- accepting it. -->
- <key>acceptDelayDistribution</key>
- <dict>
- <key>type</key>
- \
<string>contrib.performance.stats.NormalDistribution</string>
- <key>params</key>
- <dict>
- <!-- mean -->
- <key>mu</key>
- <integer>60</integer>
- <!-- standard deviation -->
- <key>sigma</key>
- <integer>60</integer>
- </dict>
- </dict>
- </dict>
- </dict>
-
Some parameters may be safely modified to suit your purposes, for example you might \
choose to disable certain profiles (by setting 'enabled' to false) in order to \
simulate only specific types of activity. Also, you can edit the params for the \
various distributions to configure how often things happen.
-Motivated readers may also develop new behaviors for existing clients, or even \
entirely new clients. An example of adding a new behavior to an existing client can \
be found here: http://trac.calendarserver.org/changeset/8428. As of this writing, we \
have only the one Snow Leopard client simulator, and would happily accept patches \
that implement additional clients! +This sim is designed to facilitate easy \
integration of new behaviors for existing clients, or even entirely new clients. An \
example of adding a new behavior to an existing client can be found here: \
http://trac.calendarserver.org/changeset/8428.
---------------------
Scalability
@@ -295,26 +317,18 @@
To use four instances on the local host::
- <key>workers</key>
- <array>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- </array>
+ <key>workers</key>
+ <array>
+ <string>./bin/python contrib/performance/loadtest/ampsim.py</string>
+ <string>./bin/python contrib/performance/loadtest/ampsim.py</string>
+ </array>
-To use four instances each on two different remote hosts, use something like::
+To use two instances each on two different remote hosts, use something like::
<key>workers</key>
<array>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string> + <string>exec ssh blade2 'cd \
~/ccs/CalendarServer ; exec ./bin/python \
contrib/performance/loadtest/ampsim.py'</string> + <string>exec ssh blade3 'cd \
~/ccs/CalendarServer ; exec ./bin/python \
contrib/performance/loadtest/ampsim.py'</string> </array>
**When using remote hosts, the ssh commands must work in an unattended fashion, so \
configure SSH keys as needed**. Also, each remote host needs to have a Calendar \
Server SVN checkout. In this example, the hosts blade2 and blade3 need to have an SVN \
checkout of Calendar Server at ~/ccs/CalendarServer. @@ -335,4 +349,3 @@
user01 - - - - - - - - - - - startup BEGIN
...
-
[Attachment #5 (text/html)]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[15439] CalendarServer/trunk/doc/Admin/LoadSimulation.rst</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: \
verdana,arial,helvetica,sans-serif; font-size: 10pt; } #msg dl a { font-weight: \
bold} #msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: \
bold; } #msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: \
6px; } #logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em \
0; } #logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg \
h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; } \
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; \
} #logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: \
-1.5em; padding-left: 1.5em; } #logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em \
1em 0 1em; background: white;} #logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid \
#fa0; border-bottom: 1px solid #fa0; background: #fff; } #logmsg table th { \
text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted \
#fa0; } #logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: \
0.2em 0.5em; } #logmsg table thead th { text-align: center; border-bottom: 1px solid \
#fa0; } #logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: \
6px; } #patch { width: 100%; }
#patch h4 {font-family: \
verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, \
#patch .copfile {border:1px solid #ccc;margin:10px 0;} #patch ins \
{background:#dfd;text-decoration:none;display:block;padding:0 10px;} #patch del \
{background:#fdd;text-decoration:none;display:block;padding:0 10px;} #patch .lines, \
.info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a \
href="http://trac.calendarserver.org//changeset/15439">15439</a></dd> <dt>Author</dt> \
<dd>dre@apple.com</dd> <dt>Date</dt> <dd>2016-01-21 15:25:21 -0800 (Thu, 21 Jan \
2016)</dd> </dl>
<h3>Log Message</h3>
<pre>Update load simulation doc</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#CalendarServertrunkdocAdminLoadSimulationrst">CalendarServer/trunk/doc/Admin/LoadSimulation.rst</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="CalendarServertrunkdocAdminLoadSimulationrst"></a>
<div class="modfile"><h4>Modified: CalendarServer/trunk/doc/Admin/LoadSimulation.rst \
(15438 => 15439)</h4> <pre class="diff"><span>
<span class="info">--- CalendarServer/trunk/doc/Admin/LoadSimulation.rst 2016-01-14 \
22:00:31 UTC (rev 15438)
+++ CalendarServer/trunk/doc/Admin/LoadSimulation.rst 2016-01-21 23:25:21 UTC (rev \
15439) </span><span class="lines">@@ -28,19 +28,8 @@
</span><span class="cx">
</span><span class="cx"> ./bin/sim
</span><span class="cx">
</span><del>-The sim should begin logging its activity, which might look something \
like: </del><ins>+The sim should begin logging its activity to stdout, a sample of \
which is shown in the next section. </ins><span class="cx">
</span><del>-::
-
- Loaded 99 accounts.
- user01 - - - - - - - - - - - startup BEGIN
- user01 request 207✓[ 0.04 s] PROPFIND /principals/users/user01/
- user01 request 207✓[ 0.02 s] PROPFIND /principals/__uids__/user01/
- user01 request 200✓[ 0.01 s] REPORT /principals/
- user01 - - - - - - - - - - - poll BEGIN
- user01 request 207✓[ 0.19 s] PROPFIND /calendars/__uids__/user01/
- ...
-
</del><span class="cx"> - To stop the sim, type control-c.
</span><span class="cx">
</span><span class="cx"> ---------------------
</span><span class="lines">@@ -56,61 +45,119 @@
</span><span class="cx">
</span><span class="cx"> ::
</span><span class="cx">
</span><del>- user01 - - - - - - - - - - - startup BEGIN
- user01 request 207✓[ 0.04 s] PROPFIND /principals/users/user01/
- user01 request 207✓[ 0.02 s] PROPFIND /principals/__uids__/user01/
- user01 request 200✓[ 0.01 s] REPORT /principals/
</del><ins>+ Loaded 99 accounts.
+ user01 - - - - - - - - - - - startup: 10.11 BEGIN
+ user02 - - - - - - - - - - - startup: 10.11 BEGIN
+ user02 request 207✓[ 0.10 s] PROPFIND{principal} \
/principals/__uids__/10000000-0000-0000-0000-000000000002/ + user01 request 207✓[ \
0.11 s] PROPFIND{principal} \
/principals/__uids__/10000000-0000-0000-0000-000000000001/ + user02 request 200✓[ \
0.03 s] REPORT{pset} /principals/ + user02 - - - - - - - - - - - poll BEGIN
+ user01 request 200✓[ 0.03 s] REPORT{pset} /principals/
</ins><span class="cx"> user01 - - - - - - - - - - - poll BEGIN
</span><del>- user01 request 207✓[ 0.29 s] PROPFIND /calendars/__uids__/user01/
- user01 request 207✓[ 0.04 s] PROPFIND /calendars/__uids__/user01/calendar/
- user01 request 207✓[ 0.04 s] PROPFIND /calendars/__uids__/user01/inbox/
- user01 request 207✓[ 0.05 s] PROPFIND /calendars/__uids__/user01/tasks/
- user01 request 207✓[ 0.02 s] PROPFIND /calendars/__uids__/user01/outbox/
- user01 request 207✓[ 0.03 s] PROPFIND /calendars/__uids__/user01/dropbox/
- user01 request 207✓[ 0.03 s] PROPFIND /calendars/__uids__/user01/notification/
- user01 - - - - - - - - - - - poll END [ 0.51 s]
- user01 - - - - - - - - - - - startup END [ 0.59 s]
</del><ins>+ user02 request 207✓[ 0.20 s] PROPFIND{home} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/ + user01 request 207✓[ \
0.20 s] PROPFIND{home} /calendars/__uids__/10000000-0000-0000-0000-000000000001/ + \
user02 request 207✓[ 0.07 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/tasks/ + user01 request \
207✓[ 0.07 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/calendar/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/tasks/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/calendar/ + user02 request \
207✓[ 0.06 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/tasks/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/calendar/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/calendar/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/tasks/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/calendar/ + user01 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/tasks/ + user02 request \
207✓[ 0.05 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/calendar/ + user02 request \
403✓[ 0.04 s] REPORT{sync-init} \
/calendars/__uids__/10000000-0000-0000-0000-000000000002/notification/ + user02 - - - \
- - - - - - - - poll END [ 0.59 s] + user02 - - - - - - - - - - - startup: 10.11 \
END [ 0.71 s] + user01 request 207✓[ 0.09 s] PROPPATCH{calendar} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/tasks/ + user01 request \
403✓[ 0.04 s] REPORT{sync-init} \
/calendars/__uids__/10000000-0000-0000-0000-000000000001/notification/ + user01 - - - \
- - - - - - - - poll END [ 0.63 s] + user01 - - - - - - - - - - - startup: 10.11 \
END [ 0.77 s] </ins><span class="cx">
</span><del>-The first token on each line is the username associated with the \
activity. Most of the time, client activity is interleaved, so to trace a specific \
client you may wish to filter for a specific username. </del><ins>+The first token on \
each line is the username associated with the activity. Note that client activity is \
highly interleaved during normal operation. </ins><span class="cx">
</span><del>-Client operations are groups of requests that are surrounded by lines \
with many dashes, either BEGIN or END. 'END' lines also include the overall duration \
of the operation, which is the elapsed wall clock time since the operation started. \
</del><ins>+Client operations are groups of requests that are delimited by lines with \
many dashes, either BEGIN or END. 'END' lines also include the overall duration of \
the operation, which is the elapsed wall clock time since the operation started. \
</ins><span class="cx"> </span><span class="cx"> Above, we see two behaviors that \
are nested; the 'startup' operation starts which includes a few requests and also \
triggers the 'poll' operation. </span><span class="cx">
</span><del>-Individual requests are logged with the word 'request', followed by the \
HTTP status, the request duration, the HTTP method of the request, and the URI that \
is being targeted. Successful requests are indicated by a ✓ while failed requests \
are indicated with —✗. </del><ins>+Individual requests are logged with the word \
'request', followed by the HTTP status code, the request duration, the HTTP method of \
the request, some additional info about the request in curly braces (for example \
'principal' refers to the target of a PROPFIND, and sync-init is a type of REPORT), \
and the URI that is being targeted. Successful requests are indicated by a ✓ while \
failed requests are indicated with —✗. </ins><span class="cx">
</span><span class="cx"> Most client operations also log a 'lag' time when they \
start, e.g. </span><span class="cx">
</span><span class="cx"> ::
</span><span class="cx">
</span><del>- user12 - - - - - - - - - - - create BEGIN {lag 0.80 ms}
</del><ins>+ user18 - - - - - - - - - - - invite BEGIN {lag 2.32 ms}
</ins><span class="cx">
</span><del>-'lag' in this context specifies the time elapsed between when the sim \
should have started this operation (based on internal timers / activity \
distributions) and when it actually did. This is intended to be a measure of client \
sim health; if the lag values begin to grow too high, that indicates that the client \
sim is 'slipping' and doesn't have enough CPU to perform its various activities at \
the rates specified in the config file. This is python, so a single instance of the \
client sim is limited to one CPU core. See the `Scalability`_ section for information \
about how to scale the load sim by combining multiple instances. </del><ins>+'lag' in \
this context specifies the time elapsed between when the sim should have started this \
operation (based on the sim configuration) and when it actually started. This is used \
as a measure of client sim health; if the lag values begin to grow too high, that \
indicates that the client sim is 'slipping' and is unable to perform its various \
activities at the rates specified in the config file. This is python 2.x, so a single \
instance of the client sim is limited to one CPU core. See the `Scalability`_ section \
for information about how to scale the load sim by combining multiple instances. \
</ins><span class="cx"> </span><span class="cx"> Summary Reports
</span><span class="cx"> ---------------
</span><span class="cx"> When the sim stops or is killed, it emits a summary report \
that displays the total number of users that were active in this run, and overall \
counts for both individual requests and also the higher-level client operations. Also \
we display quality of service information used to grade the run as PASS or FAIL. An \
example report is shown below:: </span><span class="cx">
</span><del>- Users : 20
- request count failed >3sec mean median
- DELETE 21 0 0 0.0186 0.0184
- GET 27 0 0 0.0341 0.0223
- POST 17 0 0 0.0709 0.0523
- PROPFIND 265 0 0 0.0593 0.0262
- PUT 44 0 2 0.3544 0.1735
- REPORT 107 0 0 0.0599 0.0280
</del><ins>+ ** REPORT **
+
+ * Client
+ Cpu Time : user 29.01 sys 4.34 total 00:00:33
+ Clients : 20
+ Qos : 0.7060
+ Run Time : 00:15:59
+ Start Time : 01/21 14:19:05
+ Users : 20
+
+ * Details
+ request count failed >0.1 sec >0.5 sec \
>1 sec >3 sec >5 sec >10 sec >30 sec mean median \
stddev QoS STATUS + \
-------------------------------------------------------------------------------------- \
------------------------------------------------------------------------------------ \
+ DELETE{event} 3 0 0 0 0 \
0 0 0 0 0.0715 0.0771 0.0101 0.3573 + GET \
3 0 2 0 0 0 0 0 \
0 0.1330 0.1381 0.0259 0.0000 + GET{event} \
219 0 117 1 0 0 0 0 \
0 0.1454 0.1056 0.0987 1.4541 + POST{attach} \
138 0 138 19 0 0 0 0 \
0 0.3243 0.2871 0.1502 0.0000 + POST{fb-small} \
1279 0 664 10 0 0 0 0 \
0 0.1299 0.1026 0.0807 0.2598 + POST{share-calendar} \
60 0 60 5 0 0 0 0 \
0 0.2990 0.2686 0.1206 0.0000 + PROPFIND{home} \
20 0 20 0 0 0 0 0 \
0 0.1534 0.1454 0.0204 0.6136 + PROPFIND{principal} \
20 0 1 0 0 0 0 0 \
0 0.0798 0.0829 0.0125 0.9319 + PROPPATCH{calendar} \
120 0 2 0 0 0 0 0 \
0 0.0608 0.0569 0.0124 0.6216 + PUT{attendee-medium} \
2 0 2 0 0 0 0 0 \
0 0.1585 0.1628 0.0044 0.2426 + PUT{attendee-small} \
1 0 1 1 0 0 0 0 \
0 0.6156 0.6156 0.0000 1.2312 + PUT{event} \
220 0 220 40 10 0 0 0 \
0 0.3760 0.2587 0.3363 1.5041 + PUT{organizer-large} \
7 0 6 1 0 0 0 0 \
0 0.2066 0.1865 0.1353 0.4050 + PUT{organizer-medium} \
111 0 81 8 2 0 0 0 \
0 0.2419 0.1381 0.2725 0.5354 + PUT{organizer-small} \
22 0 13 1 0 0 0 0 \
0 0.1792 0.1091 0.1762 0.7168 + PUT{update} \
78 0 78 6 0 0 0 0 \
0 0.3087 0.2923 0.1048 1.2346 + REPORT{cpsearch} \
1279 0 63 0 0 0 0 0 \
0 0.0482 0.0378 0.0385 0.0000 + REPORT{multiget-small} \
15 0 0 0 0 0 0 0 \
0 0.0510 0.0496 0.0052 0.2039 + REPORT{pset} \
20 0 0 0 0 0 0 0 \
0 0.0300 0.0302 0.0008 0.9120 + REPORT{sync-init} \
20 0 0 0 0 0 0 0 \
0 0.0427 0.0394 0.0099 0.5854 + REPORT{sync} \
15 0 0 0 0 0 0 0 \
0 0.0481 0.0471 0.0032 0.1925 +
+ operation count failed >0.1 sec >0.5 sec \
>1 sec >3 sec >5 sec >10 sec >30 sec mean median \
stddev avglag (ms) STATUS + \
-------------------------------------------------------------------------------------- \
----------------------------------------------------------------------------------------
+ accept 3 0 3 1 0 \
0 0 0 0 0.4447 0.2947 0.2365 7.8805 + \
create 220 0 220 40 10 \
0 0 0 0 0.3780 0.2615 0.3363 13.1159 + \
download 3 0 3 0 0 \
0 0 0 0 0.1515 0.1647 0.0343 0.0000 + \
invite 140 0 140 136 105 \
22 3 0 0 1.8759 1.5909 1.1846 2.3190 \
+ poll 20 20 20 20 0 \
0 0 0 0 0.6612 0.6573 0.0878 0.0000 FAIL + \
push 447 0 0 0 0 \
0 0 0 0 0.0002 0.0001 0.0004 0.0000 + \
startup: 10.11 20 0 20 20 0 \
0 0 0 0 0.7734 0.7687 0.0903 0.0000 + \
update{description} 38 0 38 19 0 \
0 0 0 0 0.5170 0.5315 0.1834 41.3110 + \
update{title} 40 0 40 17 1 \
0 0 0 0 0.5271 0.4719 0.1920 28.1850 + \
+ *** FAIL
+ Greater than 1% POLL failed
+ Exit code: 1
+
</ins><span class="cx">
</span><del>- operation count failed >3sec mean median avglag (ms)
- accept 10 0 0 0.2808 0.2942 0.8490
- create 17 0 0 0.0560 0.0484 1.0024
- invite 17 0 2 0.8713 0.4774 0.9585
- poll 64 0 0 0.3856 0.3234 0.0000
- reply done 12 0 0 0.0177 0.0174 1.9181
- startup 20 0 0 0.7369 0.6023 0.0000
- PASS
</del><span class="cx">
</span><del>-The pass / fail criteria are defined in \
`contrib/performance/loadtest/profiles.py \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/profiles.py>`_ \
for operations and `contrib/performance/loadtest/population.py \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/population.py>`_ \
for individual requests, and are generally derived from execution time and failure \
rate. </del><ins>+The pass / fail criteria are defined in \
`contrib/performance/loadtest/thresholds.json \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/thresholds.json>`_. \
This json data describes the maximum percentage ("thresholds") of each \
request and operation type that are allowed in each time bucket ("limits"), \
which if exceeded will cause that type to be failed. For example, the configuration \
for requests uses the following buckets, which correspond to the time buckets in the \
report: (values in seconds) </ins><span class="cx">
</span><ins>+``[ 0.1, 0.5, 1.0, 3.0, 5.0, 10.0, 30.0]``
+
+The PUT{event} threshold configuration states:
+
+``[ 100.0, 100.0, 100.0, 75.0, 50.0, 25.0, 0.5]``
+
+This means the PUT{event} type is considered too slow if more than 75% of them take \
longer than 3 seconds, or more than 50% take longer than 5 seconds, or more than 25% \
take longer than 10 seconds, or more than .5% take longer than 30 seconds. Setting a \
bucket to 100% effectively ignores that bucket in calculating the pass / fail \
judgement. If it seems like these values might be somewhat arbitrary, that's because \
they are. +
</ins><span class="cx"> ---------------------
</span><span class="cx"> Configuration
</span><span class="cx"> ---------------------
</span><span class="lines">@@ -129,12 +176,39 @@
</span><span class="cx"> Server Specification
</span><span class="cx"> ---------------------
</span><span class="cx">
</span><del>-Specify the URI to which the client sim should connect, e.g::
</del><ins>+The client sim supports testing of 'podded' environments. If you aren't \
using pods, not to worry, the default configuration still works. For each pod, define \
the uri and other server-specific attributes such as the amp push host and port (used \
for client notifications in lieu of APNS), and the stats socket port (if enabled in \
the server config). </ins><span class="cx">
</span><del>- <!-- Identify the server to be load tested. -->
- <key>server</key>
- <string>https://127.0.0.1:8443/</string>
</del><ins>+::
</ins><span class="cx">
</span><ins>+ <key>servers</key>
+ <dict>
+ <key>PodA</key>
+ <dict>
+ <key>enabled</key>
+ <true/>
+
+ <!-- Identify the server to be load tested. -->
+ <key>uri</key>
+ <string>https://localhost:8443</string>
+
+ <key>ampPushHosts</key>
+ <array>
+ <string>localhost</string>
+ </array>
+ <key>ampPushPort</key>
+ <integer>62311</integer>
+
+ <!-- Define whether server supports stats socket. -->
+ <key>stats</key>
+ <dict>
+ <key>enabled</key>
+ <true/>
+ <key>Port</key>
+ <integer>8100</integer>
+ </dict>
+ </dict>
+ ...
+
</ins><span class="cx"> User Accounts
</span><span class="cx"> -------------
</span><span class="cx">
</span><span class="lines">@@ -142,148 +216,96 @@
</span><span class="cx">
</span><span class="cx"> ::
</span><span class="cx">
</span><del>- <!-- Define the credentials of the clients which will \
be used to load test
- the server. These credentials must already be valid on the \
server. -->
- <key>accounts</key>
- <dict>
- <!-- The loader is the fully-qualified Python name of a \
callable which
- returns a list of directory service records defining \
all of the client accounts
- to use. \
contrib.performance.loadtest.sim.recordsFromCSVFile reads username,
- password, mailto triples from a CSV file and returns \
them as a list of faked
- directory service records. -->
- <key>loader</key>
- \
<string>contrib.performance.loadtest.sim.recordsFromCSVFile</string> \
</del><ins>+ <key>accounts</key> + <dict>
+ <!-- The loader is the fully-qualified Python name of a callable \
which + returns a list of directory service records defining all of \
the client accounts + to use. \
contrib.performance.loadtest.sim.recordsFromCSVFile reads username, + \
password, mailto triples from a CSV file and returns them as a list of faked + \
directory service records. --> + <key>loader</key>
+ <string>contrib.performance.loadtest.sim.recordsFromCSVFile</string>
</ins><span class="cx">
</span><del>- <!-- Keyword arguments may be passed to the \
loader. -->
- <key>params</key>
- <dict>
- <!-- recordsFromCSVFile interprets the path \
relative to the config.plist,
- to make it independent of the script's \
working directory while still allowing
- a relative path. This isn't a great \
solution. -->
- <key>path</key>
- \
<string>contrib/performance/loadtest/accounts.csv</string>
- </dict>
- </dict>
</del><ins>+ <!-- Keyword arguments may be passed to the loader. -->
+ <key>params</key>
+ <dict>
+ <!-- recordsFromCSVFile interprets the path relative to the \
config.plist, + to make it independent of the script's working \
directory while still allowing + a relative path. This isn't a \
great solution. --> + <key>path</key>
+ <string>contrib/performance/loadtest/accounts.csv</string>
</ins><span class="cx">
</span><ins>+ <!-- When there are accounts for multiple pods, \
interleave the accounts for each + pod so that the arrival \
mechanism will cycle clients between each pod. --> + \
<key>interleavePods</key> + <true/>
+ </dict>
+ </dict>
+
+
</ins><span class="cx"> The accounts.csv file has lines like shown below::
</span><span class="cx">
</span><del>- user01,user01,User 01,user01@example.com
</del><ins>+ user01,user01,User \
01,user01@example.com,10000000-0000-0000-0000-000000000001,PodA </ins><span \
class="cx"> </span><span class="cx"> Client Arrival
</span><span class="cx"> ----------------
</span><span class="cx">
</span><span class="cx"> This section configures the number of accounts to use, and \
defines how quickly clients are initialized when the sim starts:: </span><span \
class="cx"> </span><del>- <!-- Define how many clients will \
participate in the load test and how
- they will show up. -->
- <key>arrival</key>
- <dict>
</del><ins>+ <!-- Define how many clients will participate in the load test and \
how + they will show up. -->
+ <key>arrival</key>
+ <dict>
</ins><span class="cx">
</span><del>- <!-- Specify a class which creates new \
clients and introduces them into
- the test. \
contrib.performance.loadtest.population.SmoothRampUp introduces
- groups of new clients at fixed intervals up to a \
maximum. The size of the
- group, interval, and maximum are configured by the \
parameters below. The
- total number of clients is groups * groupSize, which \
needs to be no larger
- than the number of credentials created in the \
accounts section. -->
- <key>factory</key>
- \
<string>contrib.performance.loadtest.population.SmoothRampUp</string> \
</del><ins>+ <!-- Specify a class which creates new clients and introduces \
them into + the test. contrib.performance.loadtest.population.SmoothRampUp \
introduces + groups of new clients at fixed intervals up to a maximum. The \
size of the + group, interval, and maximum are configured by the parameters \
below. The + total number of clients is groups * groupSize, which needs to be \
no larger + than the number of credentials created in the accounts section. \
--> + <key>factory</key>
+ <string>contrib.performance.loadtest.population.SmoothRampUp</string>
</ins><span class="cx">
</span><del>- <key>params</key>
- <dict>
- <!-- groups gives the total number of groups of \
clients to introduce. -->
- <key>groups</key>
- <integer>20</integer>
</del><ins>+ <key>params</key>
+ <dict>
+ <!-- groups gives the total number of groups of clients to introduce. \
--> + <key>groups</key>
+ <integer>10</integer>
</ins><span class="cx">
</span><del>- <!-- groupSize is the number of \
clients in each group of clients. It's
- really only a "smooth" ramp up if \
this is pretty small. -->
- <key>groupSize</key>
- <integer>1</integer>
</del><ins>+ <!-- groupSize is the number of clients in each group of \
clients. It's + really only a "smooth" ramp up if this is \
pretty small. --> + <key>groupSize</key>
+ <integer>2</integer>
</ins><span class="cx">
</span><del>- <!-- Number of seconds between the \
introduction of each group. -->
- <key>interval</key>
- <integer>3</integer>
- </dict>
</del><ins>+ <!-- Number of seconds between the introduction of each \
group. --> + <key>interval</key>
+ <integer>3</integer>
</ins><span class="cx">
</span><del>- </dict>
</del><ins>+ <!-- Number of clients each user is assigned to. -->
+ <!-- Set weight of clients to 1 if this is > 1. Number of clients \
must match this value if > 1. --> + \
<key>clientsPerUser</key> + <integer>1</integer>
+ </dict>
</ins><span class="cx">
</span><del>-In the default configuration, one client is initialized every 3 seconds, \
until 20 clients are initialized. As soon as a client is initialized, it begins to \
perform its specified behaviors at the configured rates (see "Client \
Behaviors"). </del><ins>+ </dict>
</ins><span class="cx">
</span><del>-To increase the client load, increase the number of groups.
</del><span class="cx">
</span><del>-To increase the rate at which clients are initialized, reduce 'interval' \
and / or increase 'groupSize' </del><ins>+In the default configuration, three clients \
are initialized every 3 seconds, until 30 clients are initialized (groups * \
groupSize). As soon as a client is initialized, it begins to perform its specified \
behaviors at the configured rates (see "Client Behaviors"). </ins><span \
class="cx"> </span><del>-Remember: **The total number of clients is groups * \
groupSize, which needs to be no larger than the number of credentials created in the \
accounts section** </del><ins>+To increase the client load, increase the number of \
groups and / or groupSize. Take care not to exceed the number of accounts defined in \
accounts.csv. </ins><span class="cx">
</span><ins>+To increase the rate at which clients are initialized, reduce \
'interval'. +
</ins><span class="cx"> Client Behaviors
</span><span class="cx"> ----------------
</span><span class="cx">
</span><del>-The 'clients' plist key is an array of dictionaries, where each dict has \
the keys: </del><ins>+Client behaviors are defined in \
`contrib/performance/loadtest/clients.plist \
<http://trac.calendarserver.org/browser/CalendarServer/trunk/contrib/performance/loadtest/clients.plist>`_. \
The 'clients' plist key is an array of dictionaries describing the client. The \
clients.plist is well commented, so no need to repeat those details here. </ins><span \
class="cx"> </span><del>-- 'software', which specifies the implementation class for \
this client. For example: </del><ins>+- 'profiles' is an array of dictionaries \
specifying individual behaviors of each client. Each dict has a 'class' key which \
specifies the implementation class for this behavior, and a 'params' dict with \
options specific to that behavior. See the plist for more information. </ins><span \
class="cx"> </span><del>-::
-
- <key>software</key>
- <string>contrib.performance.loadtest.ical.SnowLeopard</string>
-
-- 'params', optionally specifying parameters accepted by this software. For example:
-
-::
-
- <!-- Arguments to use to initialize the SnowLeopard instance. -->
- <key>params</key>
- <dict>
- <!-- SnowLeopard can poll the calendar home at some interval. This is
- in seconds. -->
- <key>calendarHomePollInterval</key>
- <integer>30</integer>
-
- <!-- If the server advertises xmpp push, SnowLeopard can wait for \
notifications
- about calendar home changes instead of polling for them \
periodically. If
- this option is true, then look for the server advertisement for \
xmpp push
- and use it if possible. Still fall back to polling if there is no \
xmpp push
- advertised. -->
- <key>supportPush</key>
- <false />
- </dict>
-
-- 'profiles' is an array of dictionaries specifying individual behaviors of this \
software. Each dict has a 'class' key which specifies the implementation class for \
this behavior, and a 'params' dict with options specific to that behavior. For \
example:
-
-::
-
- <!-- This profile accepts invitations to events, handles cancels, and
- handles replies received. -->
- <dict>
- <key>class</key>
- <string>contrib.performance.loadtest.profiles.Accepter</string>
-
- <key>params</key>
- <dict>
- <key>enabled</key>
- <true/>
-
- <!-- Define how long to wait after seeing a new invitation \
before
- accepting it. -->
- <key>acceptDelayDistribution</key>
- <dict>
- <key>type</key>
- \
<string>contrib.performance.stats.NormalDistribution</string>
- <key>params</key>
- <dict>
- <!-- mean -->
- <key>mu</key>
- <integer>60</integer>
- <!-- standard deviation -->
- <key>sigma</key>
- <integer>60</integer>
- </dict>
- </dict>
- </dict>
- </dict>
-
</del><span class="cx"> Some parameters may be safely modified to suit your purposes, \
for example you might choose to disable certain profiles (by setting 'enabled' to \
false) in order to simulate only specific types of activity. Also, you can edit the \
params for the various distributions to configure how often things happen. \
</span><span class="cx"> </span><del>-Motivated readers may also develop new \
behaviors for existing clients, or even entirely new clients. An example of adding a \
new behavior to an existing client can be found here: \
http://trac.calendarserver.org/changeset/8428. As of this writing, we have only the \
one Snow Leopard client simulator, and would happily accept patches that implement \
additional clients! </del><ins>+This sim is designed to facilitate easy integration \
of new behaviors for existing clients, or even entirely new clients. An example of \
adding a new behavior to an existing client can be found here: \
http://trac.calendarserver.org/changeset/8428. </ins><span class="cx">
</span><span class="cx"> ---------------------
</span><span class="cx"> Scalability
</span><span class="lines">@@ -295,26 +317,18 @@
</span><span class="cx">
</span><span class="cx"> To use four instances on the local host::
</span><span class="cx">
</span><del>- <key>workers</key>
- <array>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- <string>./python contrib/performance/loadtest/ampsim.py</string>
- </array>
</del><ins>+ <key>workers</key>
+ <array>
+ <string>./bin/python \
contrib/performance/loadtest/ampsim.py</string> + \
<string>./bin/python contrib/performance/loadtest/ampsim.py</string> + \
</array> </ins><span class="cx">
</span><del>-To use four instances each on two different remote hosts, use something \
like:: </del><ins>+To use two instances each on two different remote hosts, use \
something like:: </ins><span class="cx">
</span><span class="cx"> <key>workers</key>
</span><span class="cx"> <array>
</span><del>- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec \
./python contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string>
- <string>exec ssh blade3 'cd ~/ccs/CalendarServer ; exec ./python \
contrib/performance/loadtest/ampsim.py'</string> </del><ins>+ \
<string>exec ssh blade2 'cd ~/ccs/CalendarServer ; exec ./bin/python \
contrib/performance/loadtest/ampsim.py'</string> + <string>exec ssh \
blade3 'cd ~/ccs/CalendarServer ; exec ./bin/python \
contrib/performance/loadtest/ampsim.py'</string> </ins><span class="cx"> \
</array> </span><span class="cx">
</span><span class="cx"> **When using remote hosts, the ssh commands must work in an \
unattended fashion, so configure SSH keys as needed**. Also, each remote host needs \
to have a Calendar Server SVN checkout. In this example, the hosts blade2 and blade3 \
need to have an SVN checkout of Calendar Server at ~/ccs/CalendarServer. </span><span \
class="lines">@@ -335,4 +349,3 @@ </span><span class="cx"> user01 - - - - - - - - - \
- - startup BEGIN </span><span class="cx"> ...
</span><span class="cx">
</span><del>-
</del></span></pre>
</div>
</div>
</body>
</html>
_______________________________________________
calendarserver-changes mailing list
calendarserver-changes@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-changes
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic