[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linux-rdma
Subject:    updates needed to Documentation/librdmacm.md
From:       "Rehm, Kevan Flint" <kevan.rehm () hpe ! com>
Date:       2020-03-31 22:57:05
Message-ID: EF5E5EF5-3F38-4405-A01B-3DDDC89190BC () hpe ! com
[Download RAW message or body]

Greetings,
 
Getting multiple infiniband interfaces on a node to work required more changes than \
the hints provided in Documentation/librdmacm.md.  We have a few suggestions for \
additions to that page that might save others a lot of debugging time.  
1. accept_local must be set to 1 as documented in in librdmacm.md.

2. The documentation says to set arp_ignore to the value 2.   That only works as long \
as all the interfaces on clients and servers are in the same logical subnet, e.g.   \
ib0 == 10.0.0.27, ib1 == 10.0.1.27, prefix=16.   If you happened to create separate \
logical subnets, e.g. prefix=24, then the value 2 does not work, you have to use the \
value 1.

3. The documentation does not mention the rp_filter parameter, but it must be \
modified even if you have created a single logical subnet.  The value of rp_filter \
cannot be 1, you must set it to either 0 or 2, with 2 being more secure.   The \
default value on CentOS 7 is 1.   We are using 2.  
We were surprised to find that the value of rp_filter would also get reset after a \
"systemctl restart network", not just on reboots, so the setting you want must be \
persisted in a /etc/sysctl.d file if you want the value to stay across network \
restarts.  The other parameters held their values across network restarts.  
With the above changes persisted in a new /etc/sysctl.d/95-daos.conf file, we have \
been able to successfully use multiple infiniband interfaces per node.  
Distribution is CentOS 7.
Kernel is 3.10.0-1062.18.1.el7.x86_64
 
CentOS 7 sysctl defaults:
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.rp_filter = 1
 
If you need more information, let me know,
 
Regards, Kevan


[prev in list] [next in list] [prev in thread] [next in thread]