From gentoo-user Thu Oct 03 16:57:04 2013 From: Paul Hartman Date: Thu, 03 Oct 2013 16:57:04 +0000 To: gentoo-user Subject: Re: [gentoo-user] emerge --sync issue on only one comp on LAN Message-Id: X-MARC-Message: https://marc.info/?l=gentoo-user&m=138081945217194 On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon wrote: > On 02/10/2013 19:37, Paul Hartman wrote: >> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon wrote: >>> On 01/10/2013 17:17, Greg Turner wrote: >>>> Rsync mirrors don't grow on trees, man. People pay good money to >>>> provide that service to us. You should seriously be embarrassed to >>>> have posted this. >>> >>> Really? >>> >>> >>> Then you can all use mine with the greatest of pleasure: >>> >>> SYNC="rsync://ftp.is.co.za/gentoo-portage" >>> >>> >>> I have the NetOps team BEGGING me weekly to try and generate more >>> traffic out of our network going international. The in-out ratio on the >>> peering links is seriously screwed and they badly want something to even >>> it out a bit :-) >> >> Challenge accepted. :) >> >> Here's my sync summary (I'm in the USA): >> >> Number of files: 174305 >> Number of files transferred: 50913 > ^^^^^ >> Total file size: 305.99M bytes >> Total transferred file size: 73.98M bytes >> Literal data: 73.98M bytes >> Matched data: 0 bytes >> File list size: 4.31M >> File list generation time: 343.526 seconds > ^^^^^^^^^^^^^^^ >> File list transfer time: 0.000 seconds >> Total bytes sent: 1.12M >> Total bytes received: 41.37M >> >> sent 1.12M bytes received 41.37M bytes 64.33K bytes/sec >> total size is 305.99M speedup is 7.20 > > > You don't sync very often, right? I usually sync manually daily or every other day if I'm busy and don't get a chance. I assumed there was some mass change to ebuild headers or license text or something which caused everything in the tree to get touched this week. My local portage tree is on a fast SSD in an 8-core box with 32GB of RAM and a 100mbit internet connection, so the bottleneck hopefully is not on my side of the transaction. ;) Let's do some more trials. Between yesterday and today, I have synced with my normal mirror, but I'm syncing with your server again now: Number of files: 174410 Number of files transferred: 17372 Total file size: 306.28M bytes Total transferred file size: 22.32M bytes Literal data: 22.32M bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 379.920 seconds File list transfer time: 0.000 seconds Total bytes sent: 382.35K Total bytes received: 15.71M sent 382.35K bytes received 15.71M bytes 29.33K bytes/sec total size is 306.28M speedup is 19.04 Now I'm immediately doing another sync, first deleting timestamp.chk to force it to sync again. There should be zero files to transfer (except the timestamp file). Number of files: 174410 Number of files transferred: 1 Total file size: 306.28M bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 28.612 seconds File list transfer time: 0.000 seconds Total bytes sent: 183 Total bytes received: 4.31M sent 183 bytes received 4.31M bytes 128.75K bytes/sec total size is 306.28M speedup is 71.01 Now I'm switching back to my beloved mirror.steadfast.net and running another sync. Number of files: 174409 Number of files transferred: 17364 Total file size: 306.30M bytes Total transferred file size: 21.74M bytes Literal data: 21.74M bytes Matched data: 0 bytes File list size: 4.39M File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 355.23K Total bytes received: 15.67M sent 355.23K bytes received 15.67M bytes 191.93K bytes/sec total size is 306.30M speedup is 19.11 Interestingly it transferred almost the same number of files as my first sync with yours. Comparing timestamps, your server's latest update is about 5 hours older than Steadfast's, so things must be changing frequently in portage these days! 17k changes in 5 hours... My ping to your server is 300ms, my ping to steadfast is 18ms. I don't know anything about how rsync works behind the curtain, if a higher latency would cause the file list generation to be slower, or if that is a measurement of server performance or something else. Total sync times from my log: 1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage (first sync, 17k files updated, 552 seconds) 1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage (sync with no updates except timestamp.chk, 38 seconds) 1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage (re-sync with steadfast, 17k files updated, 83 seconds) 1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage (sync with no updates except timestamp.chk, 12 seconds) HTH and thanks for the mirror :) Paul