[prev in list] [next in list] [prev in thread] [next in thread]
List: bacula-bugs
Subject: [Bacula-bugs] [bacula 0001767]: Poor bacula performance with high rtt
From: Mantis Bug Tracker <nobody () bugs ! bacula ! org>
Date: 2012-01-26 14:03:50
Message-ID: 6e571f5c4f9b160c5abdbfa3ac96a6d7 () bugs ! bacula ! org
[Download RAW message or body]
A NOTE has been added to this issue.
======================================================================
http://bugs.bacula.org/view.php?id=1767
======================================================================
Reported By: reaper
Assigned To: kern
======================================================================
Project: bacula
Issue ID: 1767
Category: File Daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: assigned
======================================================================
Date Submitted: 2011-09-30 18:09 UTC
Last Modified: 2012-01-26 14:03 UTC
======================================================================
Summary: Poor bacula performance with high rtt
Description:
I have sd in Germany and fd in USA. When backup is running bacula uses about
5-10 Mbit/s of bandwidth. With iperf test I can see traffic of about 100-120
Mbit/s. Values of Send-Q from ss taken during tests shows that iperf can
increase it up to 3MB while bacula can only do about 128kB.
I've tried to tune rmem_max/wmem_max and some other variables with no effect.
I've also tried to set "Maximum Network Buffer Size" to 8MB also with no effect.
When I created ssh tunnel between hosts and set Address = 127.0.0.1 in client
configuration backup was running at 100Mbit/s speed.
What can be done to resolve this issue?
======================================================================
----------------------------------------------------------------------
(0005984) Dan Langille (manager) - 2011-10-01 19:51
http://bugs.bacula.org/view.php?id=1767#c5984
----------------------------------------------------------------------
This appears to be a support issue. We don't deal with that here.
----------------------------------------------------------------------
(0005985) kern (administrator) - 2011-10-02 05:57
http://bugs.bacula.org/view.php?id=1767#c5985
----------------------------------------------------------------------
What Dan writes is true, this is probably more a support issue than a bug. In
any case, it is a performance problem, and I think I have previously seen a bug
report on the same subject. The difference is that the report of that bug
claimed (probably correctly) that something Bacula was doing prevented him from
setting parameters correctly.
It seems to me that if ssh can get data through faster than Bacula, there is
something I should look into. It is quite possible that it is simply
compression that ssh does that Bacula does not do.
I will setup a test environment in another week or two to see if I can reproduce
this. In addition, if you are interested in testing, please try two things:
1. Turn on compression with GZIP=9 in the FD and see if that speeds up the
transmission. Please do two controlled tests without compression, then one or
two with compression. I recommend backing up a single big file.
2. Turn off compression in Bacula, and explicitly turn off compression in ssh.
Then measure the speed difference for a given big file.
3. Use some TCP/IP tool to dump the packets that are sent by Bacula when it does
a direct connection to the SD, then do the same when you have an ssh tunnel.
Only the first minute of output will be necessary to see what happens
during the start up and the first say 1000 data transmissions.
----------------------------------------------------------------------
(0005986) kern (administrator) - 2011-10-02 05:59
http://bugs.bacula.org/view.php?id=1767#c5986
----------------------------------------------------------------------
Please specify the OS and version you are using for the Dir, SD, and FD.
----------------------------------------------------------------------
(0005988) reaper (reporter) - 2011-10-02 08:22
http://bugs.bacula.org/view.php?id=1767#c5988
----------------------------------------------------------------------
Compression does not affect backup speed. Content of backup is mostly video
files so its quite unlikely to get 10x improvement just from compression.
FD is 5.0.3 on Debian 5.0 amd64
SD/Dir is 5.0.3 on Debian 6.0 amd64
I can provide traffic dumps a bit later.
----------------------------------------------------------------------
(0005989) reaper (reporter) - 2011-10-02 18:13
http://bugs.bacula.org/view.php?id=1767#c5989
----------------------------------------------------------------------
Strange, couldn't upload dumps to tracker.
http://www.sendspace.com/filegroup/cS9J4I6sZOroAkx8cTZrnw
----------------------------------------------------------------------
(0005990) kern (administrator) - 2011-10-11 12:27
http://bugs.bacula.org/view.php?id=1767#c5990
----------------------------------------------------------------------
The dumps are probably too big to upload.
In attempting to use your download link, all I get is something that wants
me to download an iLivid.exe file (or something like that). Please find another
was to post it for download.
----------------------------------------------------------------------
(0005991) reaper (reporter) - 2011-10-11 12:39
http://bugs.bacula.org/view.php?id=1767#c5991
----------------------------------------------------------------------
you can try these links
http://i.lmn.name/tmp/dump-ext
http://i.lmn.name/tmp/dump-ssh
----------------------------------------------------------------------
(0005992) kern (administrator) - 2011-10-11 12:59
http://bugs.bacula.org/view.php?id=1767#c5992
----------------------------------------------------------------------
Well, it is a bit better, but it just sprays binary type data all over my
browser window, which is not exactly what we want. So, neither I nor my
browser (Firefox 7) know what those files are.
----------------------------------------------------------------------
(0005993) reaper (reporter) - 2011-10-11 13:02
http://bugs.bacula.org/view.php?id=1767#c5993
----------------------------------------------------------------------
wget will get you a nice file without any mess on screen :)
This is output from tcpdump -w dump. I assume this is what you want. Or you need
some other dump?
----------------------------------------------------------------------
(0006173) jau (developer) - 2012-01-26 14:03
http://bugs.bacula.org/view.php?id=1767#c6173
----------------------------------------------------------------------
Hi Reaper,
I tryed to reproduce your issue and did performance tests with and without a
tunnel, and the results where quiet similar, i did not notice a factor 10.
Direct connection :
JobId 11 Job BackupSpotlap.2012-01-25_19.44.59_07 is running.
Incremental Backup Job started: 25-Jan-12 16:45
Files=1 Bytes=11,599,872 Bytes/sec=32,221 Errors=0
Over the tunnel :
JobId 14 Job BackupSpotlap.2012-01-25_20.01.19_03 is running.
Incremental Backup Job started: 25-Jan-12 17:01
Files=1 Bytes=9,895,936 Bytes/sec=30,637 Errors=0
I tryed to download your tcpdump traces files but they are no longer avaiable.
Could you please resend them ?
Issue History
Date Modified Username Field Change
======================================================================
2011-09-30 18:09 reaper New Issue
2011-10-01 19:51 Dan Langille Note Added: 0005984
2011-10-02 05:57 kern Note Added: 0005985
2011-10-02 05:59 kern Note Added: 0005986
2011-10-02 05:59 kern Assigned To => kern
2011-10-02 05:59 kern Severity major => minor
2011-10-02 05:59 kern Status new => assigned
2011-10-02 08:22 reaper Note Added: 0005988
2011-10-02 08:22 reaper Note Edited: 0005988
2011-10-02 18:13 reaper Note Added: 0005989
2011-10-11 12:27 kern Note Added: 0005990
2011-10-11 12:27 kern Status assigned => feedback
2011-10-11 12:39 reaper Note Added: 0005991
2011-10-11 12:39 reaper Status feedback => assigned
2011-10-11 12:59 kern Note Added: 0005992
2011-10-11 13:02 reaper Note Added: 0005993
2012-01-26 14:03 jau Note Added: 0006173
======================================================================
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Bacula-bugs mailing list
Bacula-bugs@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-bugs
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic