[prev in list] [next in list] [prev in thread] [next in thread]
List: bro
Subject: [Zeek] Re: Anyone using Intel X710 NICs?
From: Michał Purzyński <michalpurzynski1 () gmail ! com>
Date: 2020-11-03 21:37:39
Message-ID: CAJ6bFK3mLxzw_pJJQwti82yTH6LmZMVgNsmwSeNeqzd6AJPXaw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hey Vlad,
it's been a while since I added those settings to the SepTUN Mark II and
the research made recently - and some things are out of date.
Intel claims it's necessary to use the hardware symmetric load-balancing.
It could be used but there's no advantage over just setting the symmetric
key, like you did.
For all traffic to be symmetrically hashed in a consistent way you need to
do what you did and I'd advise using 3-tuple - and skipping ports.
Fragmented traffic has no port numbers, so fragments further in the chain
might get incorrectly hashed. If you know you have no fragmented packets in
your monitored segment, you should be fine.
I used 'sd' everywhere for the rx-flow-hash just in case and the
distribution was good enough.
I'd also go with the smallest rx ring you can have - start with something
like 256 and monitor packet drops. Smaller ring size is way more cache
efficient and it improved the performance here.
On Tue, Nov 3, 2020 at 10:09 PM Vlad Grigorescu <vlad@es.net> wrote:
> Alright, I think we figured this out. We plan on releasing an Ansible role
> that will handle all this configuration, but the short version follows.
>
> NICs with the X710 controller have some additional features that must be
> disabled. Something like this should work for 8 workers:
>
> INF=ens2
> >
> ethtool -L $INF combined 8
> > ethtool -K $INF rxhash on
> > ethtool -K $INF ntuple on
> > set_irq_affinity 8-15 $INF # script ships with the i40e driver
> > ethtool -X $INF hkey
> > 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A
> > equal 8
> > ethtool -A $INF rx off tx off
> > ethtool -C $INF adaptive-rx off adaptive-tx off rx-usecs 125
> > ethtool -G $INF rx 1024
> > for i in rx tx tso ufo gso gro lro tx nocache copy sg txvlan rxvlan; do
> > ethtool -K $INF $i off; done
> > for proto in tcp4 udp4 tcp6 udp6; do ethtool -N $INF rx-flow-hash $proto
> > sdfn; done
> >
>
> Note that many of these settings will not persist after a reboot.
>
> On top of that, zeek-af_packet-plugin had a bug where you could not change
> the fanout mode, so you need to run version 2.1.2 or newer. These NICs
> require either FANOUT_CPU or FANOUT_QM (FANOUT_HASH could be symmetric, but
> the option to make it so is missing from ethtool).
>
> With these changes, we're seeing perfectly symmetric hashing, none of the
> bugs that plague Myricom SNFv5 cards, and increased performance compared to
> Myricom SNFv3 and SNFv5.
>
> Hope that helps someone else.
>
> --Vlad
>
> On Wed, Oct 28, 2020 at 4:23 PM Vlad Grigorescu <vlad@es.net> wrote:
> >
> > We recently purchased some Intel XXV710 NICs for our Zeek systems.
> However, symmetric hashing does not seem to work on them, at least not
> completely.
> >
> > There was some discussion here regarding adding some functionality to
> the driver to make it work, however this never landed:
> https://sourceforge.net/p/e1000/mailman/message/35199068/
> >
> > This post discusses how the X710 controller must be configured
> differently from the 82599 10G controller (used by the X520 cards):
> https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/
> >
> > The odd part is that following the SEPTun-MarkII guide[1] makes it
> *mostly* work, but we're consistently finding that ~1-2% of the traffic is
> not being symmetrically hashed. We're testing with
> can-i-use-afpacket-fanout[2] and Zeek 3.2.
> >
> > The most damning evidence is an Intel rep telling[3] a customer:
> >
> > > Unfortunately, we have been informed that the only support to setup
> symmetric RSS is via DPDK.
> >
> > Searching the mailing list archives, I found a couple of posts where
> people were encouraged to use X710-based cards, so I'm left wondering: Are
> there people using these? Are they also seeing this 1-2% asymmetry? Or am I
> missing a configuration tweak?
> >
> > Thanks,
> >
> > --Vlad
> >
> >
> > [1] - <
> https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst>
> > [2] - <https://github.com/JustinAzoff/can-i-use-afpacket-fanout>
> > [3] - <
> https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symmetric-RSS-hashing-See-quot-7-1-9-3/m-p/685998/highlight/true#M17746
>
> >
>
> --
> zeek mailing list -- zeek@lists.zeek.org
> To unsubscribe send an email to zeek-leave@lists.zeek.org
[Attachment #5 (text/html)]
<div dir="ltr"><div>Hey Vlad,</div><div><br></div><div>it's been a while since I \
added those settings to the SepTUN Mark II and the research made recently - and some \
things are out of date.</div><div><br></div><div>Intel claims it's necessary to \
use the hardware symmetric load-balancing. It could be used but there's no \
advantage over just setting the symmetric key, like you \
did.</div><div><br></div><div>For all traffic to be symmetrically hashed in a \
consistent way you need to do what you did and I'd advise using 3-tuple - and \
skipping ports.</div><div><br></div><div>Fragmented traffic has no port numbers, so \
fragments further in the chain might get incorrectly hashed. If you know you have no \
fragmented packets in your monitored segment, you should be \
fine.</div><div><br></div><div>I used 'sd' everywhere for the rx-flow-hash \
just in case and the distribution was good enough.</div><div><br></div><div>I'd \
also go with the smallest rx ring you can have - start with something like 256 and \
monitor packet drops. Smaller ring size is way more cache efficient and it improved \
the performance here.<br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Tue, Nov 3, 2020 at 10:09 PM Vlad Grigorescu <<a \
href="mailto:vlad@es.net">vlad@es.net</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr">Alright, I think we figured this \
out. We plan on releasing an Ansible role that will handle all this configuration, \
but the short version follows.<br><br>NICs with the X710 controller have some \
additional features that must be disabled. Something like this should work for 8 \
workers:<br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>INF=ens2 \
<br></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div></div><div>ethtool -L $INF combined \
8</div>ethtool -K $INF rxhash on<br>ethtool -K $INF ntuple on<br>set_irq_affinity \
8-15 $INF # script ships with the i40e driver<br>ethtool -X $INF hkey \
6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A \
equal 8<br>ethtool -A $INF rx off tx off<br>ethtool -C $INF adaptive-rx off \
adaptive-tx off rx-usecs 125<br>ethtool -G $INF rx 1024<br>for i in rx tx tso ufo gso \
gro lro tx nocache copy sg txvlan rxvlan; do ethtool -K $INF $i off; done<br>for \
proto in tcp4 udp4 tcp6 udp6; do ethtool -N $INF rx-flow-hash $proto sdfn; \
done<br></blockquote><br><div>Note that many of these settings will not persist after \
a reboot.</div><div><br></div><div>On top of that, zeek-af_packet-plugin had a bug \
where you could not change the fanout mode, so you need to run version 2.1.2 or \
newer. These NICs require either FANOUT_CPU or FANOUT_QM (FANOUT_HASH could be \
symmetric, but the option to make it so is missing from \
ethtool).<br></div><div><br></div><div>With these changes, we're seeing perfectly \
symmetric hashing, none of the bugs that plague Myricom SNFv5 cards, and increased \
performance compared to Myricom SNFv3 and SNFv5.<br></div><div><br></div><div>Hope \
that helps someone else.</div><div><br></div><div> \
--Vlad<br></div><div><br></div>On Wed, Oct 28, 2020 at 4:23 PM Vlad Grigorescu <<a \
href="mailto:vlad@es.net" target="_blank">vlad@es.net</a>> wrote:<br>><br>> \
We recently purchased some Intel XXV710 NICs for our Zeek systems. However, symmetric \
hashing does not seem to work on them, at least not completely.<br>><br>> There \
was some discussion here regarding adding some functionality to the driver to make it \
work, however this never landed: <a \
href="https://sourceforge.net/p/e1000/mailman/message/35199068/" \
target="_blank">https://sourceforge.net/p/e1000/mailman/message/35199068/</a><br>><br>> \
This post discusses how the X710 controller must be configured differently from the \
82599 10G controller (used by the X520 cards): <a \
href="https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/" \
target="_blank">https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/</a><br>><br>> \
The odd part is that following the SEPTun-MarkII guide[1] makes it *mostly* work, but \
we're consistently finding that ~1-2% of the traffic is not being symmetrically \
hashed. We're testing with can-i-use-afpacket-fanout[2] and Zeek \
3.2.<br>><br>> The most damning evidence is an Intel rep telling[3] a \
customer:<br>><br>> > Unfortunately, we have been informed that the only \
support to setup symmetric RSS is via DPDK.<br>><br>> Searching the mailing \
list archives, I found a couple of posts where people were encouraged to use \
X710-based cards, so I'm left wondering: Are there people using these? Are they \
also seeing this 1-2% asymmetry? Or am I missing a configuration \
tweak?<br>><br>> Thanks,<br>><br>> --Vlad<br>><br>><br>> [1] \
- <<a href="https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst" \
target="_blank">https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst</a>><br>> \
[2] - <<a href="https://github.com/JustinAzoff/can-i-use-afpacket-fanout" \
target="_blank">https://github.com/JustinAzoff/can-i-use-afpacket-fanout</a>><br>> \
[3] - <<a href="https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symmetric-RSS-hashing-See-quot-7-1-9-3/m-p/685998/highlight/true#M17746" \
target="_blank">https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symm \
etric-RSS-hashing-See-quot-7-1-9-3/m-p/685998/highlight/true#M17746</a>></div> \
<br>
--<br>
zeek mailing list -- <a href="mailto:zeek@lists.zeek.org" \
target="_blank">zeek@lists.zeek.org</a><br> To unsubscribe send an email to <a \
href="mailto:zeek-leave@lists.zeek.org" \
target="_blank">zeek-leave@lists.zeek.org</a></blockquote></div></div>
--
zeek mailing list -- zeek@lists.zeek.org
To unsubscribe send an email to zeek-leave@lists.zeek.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic