[prev in list] [next in list] [prev in thread] [next in thread]
List: dpdk-users
Subject: Potential bug with the X540-AT2 driver (upon missed packets), DPDK 21.11
From: Vinay Purohit <vp () cloudjuncxion ! com>
Date: 2022-01-10 19:58:50
Message-ID: 6d79604007aaa7a6f86c13104e636932 () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Summary: We've noticed that the PMD for the X540-AT2 does not recover
properly after it reports dropped packets (resulting in increments of
*rte_eth_stats.imissing*) due to CPU overload. The said driver then
continues to drop packets even when the CPU load subsides. A restart of the
just the driver (with no other changes) fixes the issue. Other drivers
(such as the I350 1GE PMD) do no exhibit this behavior under identical
traffic conditions and everything else being the same, causing us to
suspect a bug in the X540 PMD.
Here are more details:
We have a very simple application using DPDK 21.11 running on an x86_64
platform running CentOS 8. Application receives 64byte packets at 1Gbps
from port 0 of the X540-AT2 card, does some processing, and transmits
packets over port 1 of the same card. Everything's ok when the CPU load is
moderate. When processing load saturates the CPU core, the *imissing* count
increments (as expected) as PMD cannot keep up with the received packets.
The real issue is that driver continues to miss packets and increments
*imissing* *even after the CPU load subsides to levels where previously it
reported no dropped packets.* A restart of the X540 driver using
*rte_eth_dev_stop()* and *rte_eth_dev_start() *fixes the issue. Here's the
sequence:
1. CPU core moderately loaded, X540-AT2 PMD reports no missed packets
(all good).
2. CPU core saturated, PMD reports missed packets (as expected)
3. CPU core load subsides and is about the same as level in item 1
above, but PMD continues to drop packets and increments *imissing*
(strange)
4. Issue of dropped packets gets fixed after restarting the port driver
by calling rte_eth_dev_stop() and then rte_eth_dev_start() with no other
changes and no restart of the overall process/thread/application.
The above behavior is not seen with other drivers, i.e., packet drops stop
upon mitigation of the CPU load level.
Has anyone else seen the above issue with the X540-AT2 card?
Thanks,
Vinay Purohit
CloudJuncxion, Inc.
[Attachment #5 (text/html)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=us-ascii"><meta name="Generator" content="Microsoft Word 15 (filtered \
medium)"><style><!-- /* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:#1F4E79;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:702053816;
mso-list-type:hybrid;
mso-list-template-ids:2004017532 67698703 67698713 67698715 67698703 67698713 \
67698715 67698703 67698713 67698715;} @list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style></head><body lang="EN-US" link="#0563C1" vlink="#954F72" \
style="word-wrap:break-word"><div class="WordSection1"><p class="MsoNormal"><span \
style="font-size:12.0pt;color:#1f4e79">Summary: We've noticed that the PMD for the \
X540-AT2 does not recover properly after it reports dropped packets (resulting in \
increments of <i>rte_eth_stats.imissing</i>) due to CPU overload. The said driver \
then continues to drop packets even when the CPU load subsides. A restart of the just \
the driver (with no other changes) fixes the issue. Other drivers (such as the I350 \
1GE PMD) do no exhibit this behavior under identical traffic conditions and \
everything else being the same, causing us to suspect a bug in the X540 PMD. \
</span></p><p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"> \
</span></p><p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">Here are \
more details: </span></p><p class="MsoNormal"><span \
style="font-size:12.0pt;color:#1f4e79"> </span></p><p class="MsoNormal"><span \
style="font-size:12.0pt;color:#1f4e79">We have a very simple application using DPDK \
21.11 running on an x86_64 platform running CentOS 8. Application receives 64byte \
packets at 1Gbps from port 0 of the X540-AT2 card, does some processing, and \
transmits packets over port 1 of the same card. Everything's ok when the CPU load is \
moderate. When processing load saturates the CPU core, the <i>imissing</i> count \
increments (as expected) as PMD cannot keep up with the received packets. The real \
issue is that driver continues to miss packets and increments <i>imissing</i> <u>even \
after the CPU load subsides to levels where previously it reported no dropped \
packets.</u> A restart of the X540 driver using <b>rte_eth_dev_stop()</b></span><span \
style="font-size:13.0pt;font-family:"Lucida Console""> </span><span \
style="font-size:12.0pt;color:#1f4e79">and</span><span \
style="font-size:13.0pt;font-family:"Lucida Console""> </span><b><span \
style="font-size:12.0pt;color:#1f4e79">rte_eth_dev_start() </span></b><span \
style="font-size:12.0pt;color:#1f4e79">fixes the issue. Here's the \
sequence:</span><span style="font-size:13.0pt;font-family:"Lucida \
Console""> </span><span style="font-size:12.0pt;color:#1f4e79"></span></p><p \
class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"> </span></p><ol \
style="margin-top:0in" start="1" type="1"><li class="MsoListParagraph" \
style="color:#1f4e79;margin-left:0in"><span style="font-size:12.0pt">CPU core \
moderately loaded, X540-AT2 PMD reports no missed packets (all good).</span></li><li \
class="MsoListParagraph" style="color:#1f4e79;margin-left:0in"><span \
style="font-size:12.0pt">CPU core saturated, PMD reports missed packets (as \
expected)</span></li><li class="MsoListParagraph" \
style="color:#1f4e79;margin-left:0in"><span style="font-size:12.0pt">CPU core load \
subsides and is about the same as level in item 1 above, but PMD continues to drop \
packets and increments <i>imissing</i> (strange)</span></li><li \
class="MsoListParagraph" style="color:#1f4e79;margin-left:0in"><span \
style="font-size:12.0pt">Issue of dropped packets gets fixed after restarting the \
port driver by calling rte_eth_dev_stop() and then rte_eth_dev_start() with no other \
changes and no restart of the overall process/thread/application.</span></li></ol><p \
class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"> </span></p><p \
class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">The above behavior is \
not seen with other drivers, i.e., packet drops stop upon mitigation of the CPU load \
level.</span></p><p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"> \
</span></p><p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">Has \
anyone else seen the above issue with the X540-AT2 card?</span></p><p \
class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"> </span></p><p \
class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">Thanks,</span></p><p \
class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">Vinay \
Purohit</span></p><p class="MsoNormal"><span \
style="font-size:12.0pt;color:#1f4e79">CloudJuncxion, \
Inc.</span></p></div></body></html>
["smime.p7s" (application/pkcs7-signature)]
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic