[prev in list] [next in list] [prev in thread] [next in thread] 

List:       npaci-rocks-discussion
Subject:    [Rocks-Discuss] Re: 10g network with IPOIB
From:       Krishna Teja <teja781 () gmail ! com>
Date:       2017-04-26 16:27:31
Message-ID: CAOtkijHqvw=zf-evWvqm+bHHk+vz3wy1yYnsf0Xk7HUKZzqcmw () mail ! gmail ! com
[Download RAW message or body]

Hello Philip,

Thank you for the reply. I'll try it out.

I did mean an ethernet switch. My DELL ethernet switch has two SFP+ ports
that are 10G. The following is from the user manual of the switch.

Switch Ports : The Dell Networking N2024/N2024P front panel provides 24
Gigabit Ethernet (10/100/1000BASE-T) RJ-45 ports that support
auto-negotiation for speed, flow control, and duplex. The Dell Networking
N2024/N2024P models support two SFP+ 10G ports. Dell-qualified SFP+
transceivers are sold separately. Dell Networking N2000 Series switches
operate in full-duplex mode only.

The reason we are doing this is that the ports on the infiniband switch are
all full and there is no room for the new NAS box.

Krishna

On Wed, Apr 26, 2017 at 10:23 AM, Philip Papadopoulos <
ppapadopoulos@ucsd.edu> wrote:

> On Tue, Apr 25, 2017 at 11:39 AM, Krishna Teja <teja781@gmail.com> wrote:
> 
> > Hello,
> > 
> > I have recently setup a NAS box on our rocks cluster which has an
> > infiniband card in it. I want to use IPOIB to establish a 10g network
> > between the NAS box and the frontend, which has a spare ib1 interface
> > currently not being used. The idea is to use a 192.168.20.x IP addresses
> > for both the frontend and NAS box i.e., the 20 subnet and connect them
> both
> > to an ethernet switch.
> 
> ^^^^^^ You mean Infiniband switch, correct?
> 
> > I am not sure if I need to create routes and add any
> > rules in iptables for this.
> > 
> 1. create a network (rocks add network)
> 2. add an interface (e.g. ipoib)  (rocks add host interface)
> 
> Since you have IB, don't all of your compute nodes also have IB?
> 
> Here's what I just ran on a test IB cluster (rocks-75 is the frontend ,
> rocks-75-0-0 is a subordinate node
> 
> rocks add network ib subnet=192.168.20.0 netmask=255.255.255.0 mtu=2048
> servedns=true
> rocks set host interface subnet rocks-75 ib0 ib
> rocks set host interface ip rocks-75 ib0 192.168.20.1
> rocks set host interface subnet rocks-75-0-0 ib0 ib
> rocks set host interface ip rocks-75-0-0 ib0 192.168.20.254
> rocks sync config
> rocks sync host network rocks-75-0-0
> rocks sync host network rocks-75
> 
> 
> Then a quick test
> # ping -c 4 rocks-75-0-0.ib
> PING rocks-75-0-0.ib (192.168.20.254) 56(84) bytes of data.
> 64 bytes from rocks-75-0-0.ib (192.168.20.254): icmp_seq=1 ttl=64
> time=0.206 ms
> 64 bytes from rocks-75-0-0.ib (192.168.20.254): icmp_seq=2 ttl=64
> time=0.181 ms
> 64 bytes from rocks-75-0-0.ib (192.168.20.254): icmp_seq=3 ttl=64
> time=0.167 ms
> 64 bytes from rocks-75-0-0.ib (192.168.20.254): icmp_seq=4 ttl=64
> time=0.167 ms
> 
> --- rocks-75-0-0.ib ping statistics ---
> 4 packets transmitted, 4 received, 0% packet loss, time 3000ms
> rtt min/avg/max/mdev = 0.167/0.180/0.206/0.018 ms
> 
> 
> Iperf gives about 3.6Gbit/s on this link
> 
> 
> 
> 
> 
> > 
> > So my question is if this is the best way to go about it or is there some
> > better/easier way of achieving it?
> > 
> > Thanks & Regards
> > Krishna
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: http://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/
> > 20170425/ceae4fd3/attachment.html
> > 
> 
> 
> 
> --
> Philip Papadopoulos, Ph.D
> ppapadopoulos@ucsd.edu
> (858) 822-3628
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/
> 20170426/ada8aa67/attachment.html
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20170426/25a593d2/attachment.html \



[prev in list] [next in list] [prev in thread] [next in thread]