[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lustre-discuss
Subject:    [Lustre-discuss] Using Lustre on channel bonded interfaces
From:       sr () osdn ! org ! ua (Serge Ryabchun)
Date:       2006-12-17 17:17:55
Message-ID: 20061217110746.GA8968 () osdn ! org ! ua
[Download RAW message or body]

On Thu, Dec 14, 2006 at 04:31:01PM -0600, Jeremy Mann wrote:
> Felix, Evan J wrote:
> > Can you tell us more about the version of Lustre you are using, that
> > options to the bonding module are you setting, and possibly what type of
> > switch you are using?  Those things may help.    If you are able to set
> > up two separate networks on the two interfaces, it is possible(correct
> > me here guys if I have this wrong) that Lustre can just run over the two
> > interfaces and be just happy.  For example, can you make all the eth0
> > interfaces be on the 10.10.10.0/24 network, and all the eth1 interfaces
> > on the 10.10.20.0/24 network.  The just use the lnet module
> > parameters(>=1.4.6 only) to set two channels for traffic.
> 
> We are using Lustre 1.6beta5 and no options for the bonding driver and the
> switch is an HP ProCurve 2848 I wasn't aware of the magnitude of options
> for bonding and now after reading bonding.txt, I don't know what sort of
> trunking to use.

Lustre 1.6b5, linux-2.6.18, HP ProCurve 2848 and 3400 with LACP

OSS::/etc/modules.conf
options bonding mode=802.3ad xmit_hash_policy=layer3+4

1xMDS/1xGigE, 5xOSS/2xGigE/3xSW-RAID1-OST, 42xNODE/1xGigE

FILETEST/NFS/MPI of 2006/11/10 is running with following parameters:
        Unit is 1Mb = 1048576 bytes
        File size: 4096Mb to 4096Mb, buffer size: 1Mb
        Number of processes: 42, number of files: 4, total: 168
        Using direct I/O calls (read/write).
        Testing file system: /mnt/work/test
        Nodes: n01 n02 n03 n04 n05 n06 n07 n08 n09 n10 n11 n12 n13 n14 n15 n16 n17 \
n18 n19 n20 n21 n22 n23 n24 n25 n26 n27 n28 n29 n30 n31 n32 n33 n34 n35 n36 n37 n38 \
n39 n40 n41 n42

Starting test iteration = 0 for FS = 4096Mb.
--- TFS=688128Mb        WTT=1000.7s AWT= 6.0s   LWR=16.4Mb/s TWR=687.6Mb/s
--- TFS=688128Mb        RTT=626.0s ART= 3.7s    LRR=26.2Mb/s TRR=1099.2Mb/s

Testing complete in 1634.3 seconds
--- Best writing rate achieved: TWR=687.6 Mb/s (at TFS=688128Mb)
--- Best reading rate achieved: TRR=1099.2 Mb/s (at TFS=688128Mb)
Nprocs, Average TWR, Std deviation of TWR, Average TRR, Std deviation of TRR
42 687.6  0.0 1099.2  0.0

-- 
Serge Ryabchun


[prev in list] [next in list] [prev in thread] [next in thread]