[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lustre-announce
Subject:    Re: [Lustre-discuss] Lustre 1.0 performance with Gigabit Ethernet
From:       John Wagner <jw () fujitsu ! fr>
Date:       2003-12-12 22:03:58
[Download RAW message or body]

Christoph,

If you don't mind I'd like to add a little on this item since we found 
some performance issues with the Intel Pro 1000 Ethernet drivers.

Firstly it might be a good idea to see if your Gbit network is capable 
of performing at close to the 100MB/s range between two hosts. I suggest 
you use ttcp to do this. It will quickly show whether your Ethernet 
cards are set-up for maximum performance. We have found that if you are 
using Intel Pro1000 cards and the e1000 driver prior to the 5.1.13 
release  you cannot achieve more than about 50-55 MB/s over the Gbit 
link (however given your Lustre performance is lower than that then it 
would appear you have other issues involved here). However it would be 
worth getting the network interface running at the highest speed first. 
To do this with the Intel Pro 1000's (this includes the onboard NIC's on 
the most recent Intel motherboards) you will need the e1000 driver 
version 5.1.13 or above and you must set InterruptThrottleRate=0 in 
/etc/modules.conf. This is particularly useful for improving message 
passing application throughput though I don't know how much mileage it 
will give for file system based activity. However we saw ttcp improve 
from the already stated 50-55MB/s to between 92-100MB/s depending on 
your switch specification.

This will at least help get your network interconnect set-up to maximum 
speed. If your Lustre performance does not improve then the problem lies 
somewhere in the file system code or communication layer.

I you have not already known about this and it can help a little.

Regards,

John.

John Wagner
Fujitsu Systems Europe
8 rue Maryse Hilsz
31500 Toulouse
France

Christoph Biardzki wrote:

>
>
> Hello,
>
>
> I'm currently playing with Lustre 1.0 - it works really great compared 
> to earlier releases :)
>
> On two Dual-Xeon-2.8 GHz servers linked with Gigabit Ethernet I 
> configured one of them as MDS/OST and the other one as a client. The 
> "storage" is a 10k RPM SCSI disk with approx. 50 MB/s sequential write 
> bandwidth. Using Lustre I achieved around ~20 MB/s (even locally - 
> when the server mounted the filesystem). Using striping with two OST's 
> on two machines the value stays the same (around 18-20 MB/s) - both 
> disks work with ~10 MB/s.
>
> I used iozone for the tests (with and without O_DIRECT, 64k-1MB write 
> size and files from 500M to 20G). Hardly any difference except 
> O_DIRECT being slower.
>
> The NIC is an onboard Intel e1000 (I used the drivers from the lustre 
> kernel and also a current version from intel - no difference).
>
> When using two simultaneous clients total throughput goes up.
>
>
> My conclusion is that somehow the client works slowly in my 
> configuration (even on localhost). Are there any settings I could try 
> or  is this normal speed for Gigabit Ethernet connections?
>
>
>
> Thanks alot!
>
>
> - Christoph
>
>
>
>
>



_______________________________________________
Lustre-discuss mailing list
Lustre-discuss@lists.clusterfs.com
https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
[prev in list] [next in list] [prev in thread] [next in thread]