[prev in list] [next in list] [prev in thread] [next in thread]
List: haproxy
Subject: Re: HAProxy Linux kernel 2.6 tuning for high load/connections
From: xx yy <thenucker2004 () yahoo ! com>
Date: 2009-07-29 7:17:47
Message-ID: 959837.33852.qm () web57604 ! mail ! re1 ! yahoo ! com
[Download RAW message or body]
From: Benoit <maverick@maverick.eu.org>
> Why so few memory/cpu for apache which is the most memory/cpu hog of the
> two webservers you have ?
My mistake, I wanted to benchmark Apache too but I quited now.
> Well, did you bench your webservers first ? on the loopback ?
> Before playing with system settings the most effective mean to change
> the perfs of a webserver is to tweak the webserver.
I benchmarked on the loopback lighttpd and look what happens :
root@lighttpdubu:~# httperf --server 127.0.0.1 --port 80 --uri /index.html --rate \
250 --num-conn 10000 --num-call 1 --timeout 5 httperf --timeout=5 --client=0/1 \
--server=127.0.0.1 --port=80 --uri=/index.html --rate=250 --send-buffer=4096 \
--recv-buffer=16384 --num-conns=10000 --num-calls=1 Maximum connect burst length: 8
Total: connections 10000 requests 10000 replies 10000 test-duration 39.997 s
Connection rate: 250.0 conn/s (4.0 ms/conn, <=9 concurrent connections)
Connection time [ms]: min 0.1 avg 0.2 max 31.3 median 0.5 stddev 1.5
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000
Request rate: 250.0 req/s (4.0 ms/req)
Request size [B]: 70.0
Reply rate [replies/s]: min 250.0 avg 250.0 max 250.0 stddev 0.0 (7 samples)
Reply time [ms]: response 0.2 transfer 0.0
Reply size [B]: header 214.0 content 46.0 footer 0.0 (total 260.0)
Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 34.66 system 5.30 (user 86.7% system 13.3% total 99.9%)
Net I/O: 80.6 KB/s (0.7*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
======= It ends without time outs ============
But lest do it one more time, but this time we will let it run a little longer
root@lighttpdubu:~# httperf --server 127.0.0.1 --port 80 --uri /index.html --rate \
250 --num-conn 100000 --num-call 1 --timeout 5 httperf --timeout=5 --client=0/1 \
--server=127.0.0.1 --port=80 --uri=/index.html --rate=250 --send-buffer=4096 \
--recv-buffer=16384 --num-conns=100000 --num-calls=1 Maximum connect burst length: 8
Total: connections 82074 requests 49068 replies 49068 test-duration 356.906 s
Connection rate: 230.0 conn/s (4.3 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 0.1 avg 608.0 max 3048.0 median 0.5 stddev 1204.7
Connection time [ms]: connect 607.8
Connection length [replies/conn]: 1.000
Request rate: 137.5 req/s (7.3 ms/req)
Request size [B]: 70.0
Reply rate [replies/s]: min 0.4 avg 138.2 max 250.8 stddev 124.3 (71 samples)
Reply time [ms]: response 0.2 transfer 0.0
Reply size [B]: header 214.0 content 46.0 footer 0.0 (total 260.0)
Reply status: 1xx=0 2xx=49068 3xx=0 4xx=0 5xx=0
CPU time [s]: user 76.52 system 280.30 (user 21.4% system 78.5% total 100.0%)
Net I/O: 44.3 KB/s (0.4*10^6 bps)
Errors: total 39141 client-timo 31987 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 7154 addrunavail 0 ftab-full 0 other 0
====== I had to stop it because it timed out with the same settings ==========
So this is just a matter of time until it times out because the tcp ports do not \
reuse, this is why I need some advices on tuning the system stack.
This is what I have until now:
net.ipv4.tcp_fin_timeout = 1
net.ipv4.ip_local_port_range="1024 65536"
sysctl -w fs.file-max=2000000
Are my tests wrong or the system can't handle more than 200 requests/second ?
> You also need to adjust haproxy settings according to your webservers
> capabilities (maxconn/weight/minconn/inter).
Can you please exemplify this ?
> A basic apache2 memory imprint is around 10Mb/process, so to answer 100
> concurrent connections it need 1Go of memory
This is why I quited Apache. I am looking for a testing methodology in order
to determine the maximum request/second that a server can handle - I know it is not
HAProxy fully related but any suggestions are very welcomed.
> I would recommend tuning the software stack *before* tuning the system
> stack.
I followed your advice and I tuned the webservers a little changing to epoll, \
sendfile and noatime, but Willy Thereau did not posted his sysctl parameters and \
HAProxy settings used in the last test and I find it hard to beleive that a 2.6 \
kernel can handle 38000 concurrent connections without at least increasing the \
tcp_backlog queue or decreasing the fin_timeout. Mr Thereau can you please answer to \
this? :)
[Attachment #3 (text/html)]
<html><head><style type="text/css"><!-- DIV {margin:0px;} \
--></style></head><body><div \
style="font-family:arial,helvetica,sans-serif;font-size:10pt"><div> </div><div \
style="font-family: arial,helvetica,sans-serif; font-size: 10pt;"><font face="Tahoma" \
size="2"><b><span style="font-weight: bold;">From:</span></b> Benoit \
<maverick@maverick.eu.org></font><br><br>> Why so few memory/cpu for apache \
which is the most memory/cpu hog of the<br>> two webservers you have ?<br><br>My \
mistake, I wanted to benchmark Apache too but I quited now.<br><br>> Well, did you \
bench your webservers first ? on the loopback ?<br>> Before playing with system \
settings the most effective mean to change<br>> the perfs of a webserver is to \
tweak the webserver.<br><br>I benchmarked on the loopback lighttpd and look what \
happens :<br><br>root@lighttpdubu:~# httperf --server 127.0.0.1 --port 80 --uri \
/index.html --rate 250 --num-conn 10000 --num-call 1
--timeout 5<br>httperf --timeout=5 --client=0/1 --server=127.0.0.1 --port=80 \
--uri=/index.html --rate=250 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 \
--num-calls=1<br>Maximum connect burst length: 8<br><br>Total: connections 10000 \
requests 10000 replies 10000 test-duration 39.997 s<br><br>Connection rate: 250.0 \
conn/s (4.0 ms/conn, <=9 concurrent connections)<br>Connection time [ms]: min 0.1 \
avg 0.2 max 31.3 median 0.5 stddev 1.5<br>Connection time [ms]: connect \
0.0<br>Connection length [replies/conn]: 1.000<br><br>Request rate: 250.0 req/s (4.0 \
ms/req)<br>Request size [B]: 70.0<br><br>Reply rate [replies/s]: min 250.0 avg 250.0 \
max 250.0 stddev 0.0 (7 samples)<br>Reply time [ms]: response 0.2 transfer \
0.0<br>Reply size [B]: header 214.0 content 46.0 footer 0.0 (total 260.0)<br>Reply \
status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0<br><br>CPU time [s]: user 34.66 system 5.30 \
(user 86.7% system 13.3% total 99.9%)<br>Net I/O: 80.6 KB/s (0.7*10^6 \
bps)<br><br>Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset \
0<br>Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0<br><br>======= It ends \
without time outs ============<br><br>But lest do it one more time, but this time we \
will let it run a little longer<br><br>root@lighttpdubu:~# httperf --server \
127.0.0.1 --port 80 --uri /index.html --rate 250 --num-conn 100000 --num-call 1 \
--timeout 5<br>httperf --timeout=5 --client=0/1 --server=127.0.0.1 --port=80 \
--uri=/index.html --rate=250 --send-buffer=4096 --recv-buffer=16384 \
--num-conns=100000 --num-calls=1<br>Maximum connect burst length: 8<br><br>Total: \
connections 82074 requests 49068 replies 49068 test-duration 356.906 \
s<br><br>Connection rate: 230.0 conn/s (4.3 ms/conn, <=1022 concurrent \
connections)<br>Connection time [ms]: min 0.1 avg 608.0 max 3048.0 median 0.5 stddev \
1204.7<br>Connection time [ms]: connect 607.8<br>Connection length [replies/conn]: \
1.000<br><br>Request rate: 137.5 req/s (7.3 ms/req)<br>Request size [B]: \
70.0<br><br>Reply rate [replies/s]: min 0.4 avg 138.2 max 250.8 stddev 124.3 (71 \
samples)<br>Reply time [ms]: response 0.2 transfer 0.0<br>Reply size [B]: header \
214.0 content 46.0 footer 0.0 (total 260.0)<br>Reply status: 1xx=0 2xx=49068 3xx=0 \
4xx=0 5xx=0<br><br>CPU time [s]: user 76.52 system 280.30 (user 21.4% system 78.5% \
total 100.0%)<br>Net I/O: 44.3 KB/s (0.4*10^6 bps)<br><br>Errors: total 39141 \
client-timo 31987 socket-timo 0 connrefused 0 connreset 0<br>Errors: fd-unavail 7154 \
addrunavail 0 ftab-full 0 other 0<br><br>====== I had to stop it because it timed out \
with the same settings ==========<br><br>So this is just a matter of time until it \
times out because the tcp ports do not reuse,<br>this is why I need some advices on \
tuning the system stack.<br><br>This is what I have until \
now:<br><br>net.ipv4.tcp_fin_timeout = 1<br>net.ipv4.ip_local_port_range="1024 \
65536"<br>sysctl -w fs.file-max=2000000<br><br>Are my tests wrong or the system can't \
handle more than 200 requests/second ?<br><br><br>> You also need to adjust \
haproxy settings according to your webservers<br>> capabilities \
(maxconn/weight/minconn/inter).<br><br>Can you please exemplify this ?<br><br>> A \
basic apache2 memory imprint is around 10Mb/process, so to answer 100<br>> \
concurrent connections it need 1Go of memory<br><br>This is why I quited Apache. I am \
looking for a testing methodology in order<br>to determine the maximum request/second \
that a server can handle - I know it is not<br>HAProxy fully related but any \
suggestions are very welcomed.<br><br>> I would recommend tuning the software \
stack *before* tuning the system<br>> stack.<br><br>I followed your advice and I \
tuned the webservers a little changing to epoll, sendfile<br>and noatime, but Willy \
Thereau did not posted his sysctl parameters and HAProxy <br>settings used in the \
last test and I find it hard to beleive that a 2.6 kernel can<br>handle 38000 \
concurrent connections without at least increasing the tcp_backlog queue<br>or \
decreasing the fin_timeout. Mr Thereau can you please answer to this? \
:)<br><br><br><br><br><div style="font-family: arial,helvetica,sans-serif; font-size: \
13px;"><br></div></div></div><br>
</body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic