[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: [Lustre-discuss] Lustre-1.8.1.1 over o2ib gives Input/Output error while executing lctl ping
From: vipul () chelsio ! com (Vipul Pandya)
Date: 2010-02-26 8:56:13
Message-ID: 8A71B368A89016469F72CD08050AD334076F368A () maui ! asicdesigners ! com
[Download RAW message or body]
Hi Issac,
This was very helpful. Thanks a lot for your response.
Vipul
-----Original Message-----
From: He.Huang@Sun.COM [mailto:He.Huang at Sun.COM]
Sent: 26 February 2010 02:27
To: Vipul Pandya
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Lustre-1.8.1.1 over o2ib gives
Input/Output error while executing lctl ping
On Mon, Feb 22, 2010 at 03:22:52AM -0800, Vipul Pandya wrote:
> Hello Issac,
Hi Vipul,
> ......
> I lowered the map_on_demand value to 16 and now it works fine.
>
> However, I had once concern, whether lowering down this map_on_demand
> value would impact the performance of Lustre or not?
For iWARP, you probably have no alternative. I remembered that there's
a restriction somewhere in the iWARP stack that limits the size of SQs
(which was why the rdma_create_qp errors happened), and lowering
map_on_demand is the only way to reduce Lustre SQ length.
For infiniband, lowering map_on_demand essentially reduces the # of
RDMA WQE needed for each Lustre bulk data movement, at the cost of
memory registration/deregistration at most per bulk transfer; without
map_on_demand the o2iblnd uses a static MR so there's no memory
registration cost. There could a point in the # of frags of the bulk
buffer, where the cost of handling RDMA WQEs (which usually equals
the # of frags) exceeds the cost of MR, and that's what you should
set map_on_demand to. However, since both costs are mostly determined
by HCA hardware/firmware implementation, there's no one good setting
for all interconnects, and you can only find it by testing. The LNet
selftest is a useful tool for running such tests:
http://manual.lustre.org/manual/LustreManual16_HTML/LustreIOKit.html#506
10302_36273
Hope this helps,
Isaac
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic