[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: Re: [lustre-discuss] EXT SENDER- Mounting Lustre over IB-to-Ethernet gateway
From: "Kevin M. Hildebrand" <kevin () umd ! edu>
Date: 2016-08-01 12:23:14
Message-ID: CAJmU7Qm-pwoG40A6KeXAv+n54gE61ZH7W21ij7wbb7Hh+T-jTQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
At the moment, the Lustre servers only have a low-bandwidth (1G) management
network that would be insufficient for running Lustre. The IB gateway
theoretically gives me 40G Ethernet connections to my clients which also
have 40G Ethernet ports. I could potentially set up a Lustre router, but
would like to use the hardware I already have in place, if possible.
Thanks,
Kevin
On Mon, Aug 1, 2016 at 7:58 AM, Stolarek, Marcin <marcin.stolarek@delphi.com
> wrote:
> Personally I'd suggest setting up lnet router between tcp and lustre
> network instead of using mellanox ib 2 ethernet gateway. It may be even
> easier if you have ethernet on lustre oss/mds servers but you haven't
> configured lnet for tcp?
>
>
>
> As far as I know this gateway works as arp proxy, which can lead you to
> really interesting and uncommon scenarios. This will increase the time
> needed to find the solution when incident happens J
>
>
>
> Cheers,
>
> Marcin
>
>
>
> *From:* lustre-discuss [mailto:lustre-discuss-bounces@lists.lustre.org] *On
> Behalf Of *Kevin M. Hildebrand
> *Sent:* Monday, August 01, 2016 1:06 PM
> *To:* lustre-discuss@lists.lustre.org
> *Subject:* EXT SENDER-[lustre-discuss] Mounting Lustre over
> IB-to-Ethernet gateway
>
>
>
> Our Lustre filesystem is currently set up to use the o2ib interface only-
> all of the servers have
> options lnet networks=o2ib0(ib0)
>
> We've just added a Mellanox IB-to-Ethernet gateway and would like to be
> able to have clients on the Ethernet side also mount Lustre. The gateway
> extends the same layer-2 IP range that's being used for IPoIB out to the
> Ethernet clients
>
> How should I go about doing this? Since the clients don't have IB, it
> doesn't appear that I can use o2ib0 to mount. Do I need to add another
> lnet network on the servers? Something like
> options lnet networks=o2ib0(ib0),tcp0(ib0)? Can I have both protocols on
> the same interface?
>
> And if I do have to add another lnet network, is there any way to do so
> without restarting the servers?
>
>
>
> Thanks,
>
> Kevin
>
>
> --
>
> Kevin Hildebrand
>
> University of Maryland, College Park
>
> Division of IT
>
>
> ****************************************************************************************
> Note: If the reader of this message is not the intended recipient, or an
> employee or agent responsible for delivering this message to the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this communication in error, please notify us immediately by replying to
> the message and deleting it from your computer. Thank you.
> ****************************************************************************************
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div class="gmail_default" style="font-size:small">At the moment, the \
Lustre servers only have a low-bandwidth (1G) management network that would be \
insufficient for running Lustre. The IB gateway theoretically gives me 40G Ethernet \
connections to my clients which also have 40G Ethernet ports. I could potentially \
set up a Lustre router, but would like to use the hardware I already have in place, \
if possible. <br><br></div><div class="gmail_default" \
style="font-size:small">Thanks,<br></div><div class="gmail_default" \
style="font-size:small">Kevin<br></div></div><div class="gmail_extra"><br><div \
class="gmail_quote">On Mon, Aug 1, 2016 at 7:58 AM, Stolarek, Marcin <span \
dir="ltr"><<a href="mailto:marcin.stolarek@delphi.com" \
target="_blank">marcin.stolarek@delphi.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Personally \
I'd suggest setting up lnet router between tcp and lustre network instead of using \
mellanox ib 2 ethernet gateway. It may be even easier if you have ethernet on lustre \
oss/mds servers but you haven't configured lnet for tcp?<u></u><u></u></span></p> \
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> \
<u></u></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As \
far as I know this gateway works as arp proxy, which can lead you to really \
interesting and uncommon scenarios. This will increase the time needed to find the \
solution when incident happens </span><span \
style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">J</span><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> \
<u></u></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Cheers,<u></u><u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Marcin \
<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> \
<u></u></span></p> <p class="MsoNormal"><b><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> \
lustre-discuss [mailto:<a href="mailto:lustre-discuss-bounces@lists.lustre.org" \
target="_blank">lustre-discuss-bounces@lists.lustre.org</a>] <b>On Behalf Of \
</b>Kevin M. Hildebrand<br> <b>Sent:</b> Monday, August 01, 2016 1:06 PM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" \
target="_blank">lustre-discuss@lists.lustre.org</a><br> <b>Subject:</b> EXT \
SENDER-[lustre-discuss] Mounting Lustre over IB-to-Ethernet \
gateway<u></u><u></u></span></p> <p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Our Lustre filesystem is currently \
set up to use the o2ib interface only- all of the servers have<br> options lnet \
networks=o2ib0(ib0)<u></u><u></u></p> </div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">We've just added a Mellanox \
IB-to-Ethernet gateway and would like to be able to have clients on the Ethernet side \
also mount Lustre. The gateway extends the same layer-2 IP range that's being \
used for IPoIB out to the Ethernet clients<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">How should I go about doing this? Since the clients don't \
have IB, it doesn't appear that I can use o2ib0 to mount. Do I need to add \
another lnet network on the servers? Something like<br> options lnet \
networks=o2ib0(ib0),tcp0(ib0)? Can I have both protocols on the same \
interface?<u></u><u></u></p> </div>
<div>
<p class="MsoNormal">And if I do have to add another lnet network, is there any way \
to do so without restarting the servers?<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kevin<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
--<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kevin Hildebrand<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">University of Maryland, College Park<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Division of IT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
**************************************************************************************** \
Note: If the reader of this message is not the intended recipient, or an employee or \
agent responsible for delivering this message to the intended recipient, you are \
hereby notified that any dissemination, distribution or copying of this \
communication is strictly prohibited. If you have received this communication in \
error, please notify us immediately by replying to the message and deleting it from \
your computer. Thank you.
****************************************************************************************
</div>
</blockquote></div><br></div>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic