[prev in list] [next in list] [prev in thread] [next in thread]
List: e1000-devel
Subject: Re: [E1000-devel] RSS is not supported on Intel 82599 under
From: "Ronciak, John" <john.ronciak () intel ! com>
Date: 2010-04-05 22:47:41
Message-ID: DDC57477F5D6F845A0DDCB99D3C4812D0C9811C7CF () orsmsx510 ! amr ! corp ! intel ! com
[Download RAW message or body]
Ocean,
MSI-X is “Enable+” in the following ‘lspci’ \
output. Does this mean that MSI-X is already enabled(IXGBE_FLAG_RSS_ENABLED) on this \
device or just MSI-X is support by the device(IXGBE_FLAG_RSS_CAPABLE)? Thanks!
It just means that it is capable of supporting MSI-X not that it is being used to \
communicate with the device. This is true for all the information from supplied by \
'lspci'.
Cheers,
John
-----------------------------------------------------------
"...that your people will judge you on what you can build, not what you destroy.", B. \
Obama, 2009
________________________________
From: Ocean.Ma@Emerson.com [mailto:Ocean.Ma@Emerson.com]
Sent: Saturday, April 03, 2010 12:45 AM
To: Ronciak, John; Zhang, Guanghua
Cc: e1000-devel@lists.sourceforge.net; Robertrosser@ami.com
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI
John
Referring the articles and thread here,
http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg01285.html and \
http://download.intel.com/design/network/applnots/319935.pdf
MSI-X is “Enable+” in the following ‘lspci’ \
output. Does this mean that MSI-X is already enabled(IXGBE_FLAG_RSS_ENABLED) on this \
device or just MSI-X is support by the device(IXGBE_FLAG_RSS_CAPABLE)? Thanks!
root@localhost:~# lspci -vvv -s 05:00.0
05:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit Network Connection \
(rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- \
SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- \
<MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at fbe20000 (64-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at c020 [size=32]
Region 4: Memory at fbe44000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA \
PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0 \
Enable- Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <512ns, L1 <64us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 256 bytes, MaxReadReq 256 bytes
Link: Supported Speed unknown, Width x8, ASPM L0s, Port 0
Link: Latency L0s <1us, L1 <8us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed unknown, Width x8
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 4c-e4-2e-ff-ff-af-01-00
Capabilities: [150] Unknown (14)
Capabilities: [160] Unknown (16)
Best Regards,
Ocean
________________________________
From: Ronciak, John [mailto:john.ronciak@intel.com]
Sent: 2010年4月3日 1:39
To: Ma, Ocean [NETPWR/EMBCO/CN]; Zhang, Guanghua
Cc: e1000-devel@lists.sourceforge.net; Robertrosser@ami.com
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI
This still gets back to have MSI-X interrupts enable and working in Linux. If \
/proc/interrupts is not showing that, none of the things you are trying to get \
working with our driver are going to work. It should like the HW is capable but if \
the OS is not configured or interpreting the BIOS tables wrong, MSI-X isn't going to \
work.
Cheers,
John
-----------------------------------------------------------
"...that your people will judge you on what you can build, not what you destroy.", B. \
Obama, 2009
________________________________
From: Ocean.Ma@Emerson.com [mailto:Ocean.Ma@Emerson.com]
Sent: Friday, April 02, 2010 3:39 AM
To: Zhang, Guanghua; Ronciak, John
Cc: e1000-devel@lists.sourceforge.net; Robertrosser@ami.com
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI John, Guanghua and Robert
Thanks a lot for your time!
Here is another info upon this issue. Hope it’s helpful.
Using the same Hardware, the same BIOS, we installed Windows2008 Server R2 \
on this board, and then install the 82599 NIC driver.
From windows, it seems the RSS works OK.Check the attached snapshots.
Thanks again. I am keeping check linux kernel codes and will update you in \
time.
Best Regards,
Ocean
________________________________
From: Zhang, Guanghua [mailto:guanghua.zhang@intel.com]
Sent: 2010年4月2日 14:12
To: Ma, Ocean [NETPWR/EMBCO/CN]; Ronciak, John
Cc: e1000-devel@lists.sourceforge.net; Robertrosser@ami.com; Zhang, Guanghua
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI
Ocean,
Can you still find the following information in dmesg log:
ixgbe: Multiple Queue Support Enabled
ixgbe: 0000:05:00.0: ixgbe_check_options: RSS is not supported on this hardware. \
Disabling RSS. …..
ixgbe: 0000:05:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count \
= 1, Tx Queue count = 1
If so:
I have checked the source code of ixgbe, it will happen only in ixgbe_param.c:
….
if (*aflags & IXGBE_FLAG_RSS_ENABLED) {
if (!(*aflags & IXGBE_FLAG_RSS_CAPABLE)) {
DPRINTK(PROBE, INFO,
"RSS is not supported on this "
"hardware. Disabling RSS.\n");
*aflags &= ~IXGBE_FLAG_RSS_ENABLED;
feature[RING_F_RSS].indices = 0;
} ….
That is IXGBE_FLAG_RSS_CAPABLE is not set in flag, which is set in ixgbe_main.c:
#ifdef IXGBE_RSS
if (adapter->flags & IXGBE_FLAG_MQ_CAPABLE)
{
printk(KERN_ERR " ==> ixgbe RSS flag is set \n");
adapter->flags |= IXGBE_FLAG_RSS_CAPABLE;
}
#endif
Can you track the source code as above? Let check if IXGBE_RSS is defined and \
IXGBE_FLAG_MQ_CAPABLE is set.
Thanks & Regards,
Guanghua
________________________________
From: Ocean.Ma@Emerson.com [mailto:Ocean.Ma@Emerson.com]
Sent: 2010年4月1日 21:40
To: Ronciak, John
Cc: e1000-devel@lists.sourceforge.net; Robertrosser@ami.com; Zhang, Guanghua
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI
John
Thanks agian for your time on this mail.
We are using Linux 2.6.27.18, built against the attached kernel.cfg I sent in \
previous email. This is based on OSV of WindRiver PNE Linux. And the HW is a self \
built board based on Tylersburg 5520 chipset+ single Xeon Westmere CPU with serveral \
82576 and two 82599 populated.
Options CONFIG_ARCH_SUPPORTS_MSI=y,CONFIG_PCI_MSI=y have been set in the kernel \
building config file. Just being curious at and trying to find out what else options \
should also be enabled switch on MSI-X interrupt mode.
At Linux booting phase,Linux has "Setting APIC routing to physical flat" mode...
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x10] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x12] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0b] disabled)
ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 0, version 0, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Setting APIC routing to physical flat
ACPI: HPET id: 0x8086a301 base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 12 CPUs, 6 hotplug CPUs
mapped APIC to ffffffffff5fc000 ( fee00000)
mapped IOAPIC to ffffffffff5fb000 (00000000fec00000)
Digging into the 2.6.27 kernel source codes, the API of setup_apic_routing will check \
the APIC IDs in bios_cpu_apicid and choose the APIC mode. void __init \
setup_apic_routing(void) {
if (uv_system_type == UV_NON_UNIQUE_APIC)
genapic = &apic_x2apic_uv_x;
else
#ifdef CONFIG_ACPI //( yes, we configured ACPI in kernel.cfg)
/*
* Quirk: some x86_64 machines can only use physical APIC mode
* regardless of how many processors are present (x86_64 ES7000
* is an example).( BTW, how about Westmere??)
*/
/* this "if" condition should be false. we checked the BIOS FADT part revision \
and flags.ACPI_FADT_APIC_PHYSICAL not specified by the BIOS FADT flag */ if \
(acpi_gbl_FADT.header.revision > FADT2_REVISION_ID && (acpi_gbl_FADT.flags & \
ACPI_FADT_APIC_PHYSICAL)) genapic = &apic_physflat;
else
#endif
/* you can refer to the booting log, it seems bios set lapci_id for some cores with a \
value bigger than 8 even there are only 6 cores. So this the BIOS problem? -- */ if \
(max_physical_apicid < 8) genapic = &apic_flat;
else
genapic = &apic_physflat;
printk(KERN_INFO "Setting APIC routing to %s\n", genapic->name);
}
Best Regards,
Ocean
________________________________
From: Ronciak, John [mailto:john.ronciak@intel.com]
Sent: 2010年4月1日 0:08
To: Ma, Ocean [NETPWR/EMBCO/CN]
Cc: e1000-devel@lists.sourceforge.net
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI Ocean,
We will use this mail list and not the Intel community forum for this issue. So only \
please update on this thread.
What OS is this? Is it a kernel.org kernel or a release from an OSV? Is this \
non-production HW?
The interrupts are clearly not MSI-X as shown in the table. So that need to be fixed \
for this system somehow. I'm not sure what kind of system HW this is but maybe there \
is some sort of BIOS setting that is limiting the interrupt setting? The igb driver \
is reporting MSI-X by in the same log trace it says it's using a legacy interrupt. \
Something strange is going on there. The interrupt table is clearly showing that the \
system is not using MSI-X. Without MSI-X none of the things you are trying to get to \
work will work unless MSI-X interrupts are being enabled and used.
BTW, MSI != MSI-X. With MSI all of the interrupts as still sent to a single CPU. \
Only using MSI-X can they be separated to different CPU's.
Cheers,
John
-----------------------------------------------------------
"...that your people will judge you on what you can build, not what you destroy.", B. \
Obama, 2009
________________________________
From: Ocean.Ma@Emerson.com [mailto:Ocean.Ma@Emerson.com]
Sent: Tuesday, March 30, 2010 7:20 PM
To: Ronciak, John
Cc: e1000-devel@lists.sourceforge.net
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI
John
Thanks so much of your quick reply!! I noticed your reply in Intel community.
MSI-X should have been enabled. As we also have 82576 populated on the same \
board( based on Tylerburg 5520 + 1 Westmere CPU), 82576’s multi-queue could \
successfully be enabled. As your suggestion, attached log contains the interrupt \
routing table of my board.
My 82576 populated on this board has bound 2 interfaces named by base1,base2 and \
my 82599 populated on this board named with fabric1 and fabric2.
The system’s interrupt routing mode has been set as “physical \
flat” at Linux booting up phase(with boot log: “Setting APIC routing to \
physical flat”” ACPI: Using IOAPIC for interrupt routing”). All the \
non-system IRQs have been routed just to core CPU0 :(
Maybe my issue is caused by the IRQ routing mode of my Board/Linux? If yes, there \
must be some kernel options I did not configure properly?
> From the readme file of ixgbe driver
-----------------------------------------------------------------------------------------
“
RSS - Receive Side Scaling (or multiple queues for receives)
……
RSS also effects the number of transmit queues allocated on 2.6.23 and
newer kernels with CONFIG_NETDEVICES_MULTIQUEUE set in the kernel .config file.
CONFIG_NETDEVICES_MULTIQUEUE only exists from 2.6.23 to 2.6.26. Other options
enable multiqueue in 2.6.27 and newer kernels.”
-----------------------------------------------------------------------------------------
Does someone know what “Other options” should I enable to make \
multiqueue work in a 2.6.27 kernel?.
Pls check for details from my attached log snippets.
Thanks again!
Best Regards,
Ocean
-----Original Message-----
From: Ronciak, John [mailto:john.ronciak@intel.com]
Sent: 2010年3月31日 7:06
To: Ma, Ocean [NETPWR/EMBCO/CN]; e1000-devel@lists.sourceforge.net
Subject: RE: [E1000-devel] RSS is not supported on Intel 82599 under Linux2.6.27.18, \
driver version 2.0.44.14-NAPI
Did you read my reply to the other Intel Community posting you did?
Here is the content of that reply:
"My guess is that the preempt kernel that is being used is not using MSI-X \
interrupts. If you look in the trace there is an error message that things are being \
disabled due to the interrupt type. So pick a normal kernel and things should work \
for you. You can verify if MSI-X interrupts are not being used by doing a 'cat \
/proc/interrupts'."
The 82599 driver does install on 2.6.27. I don't think that we tried .18 \
specifically but that's not the problem you are describing in al of your posts.
Cheers,
John
-----------------------------------------------------------
"...that your people will judge you on what you can build, not what you destroy.", B. \
Obama, 2009
> -----Original Message-----
> From: Ocean.Ma@Emerson.com [mailto:Ocean.Ma@Emerson.com]
> Sent: Tuesday, March 30, 2010 7:21 AM
> To: e1000-devel@lists.sourceforge.net
> Cc: Ocean.Ma@Emerson.com
> Subject: [E1000-devel] RSS is not supported on Intel 82599
> under Linux 2.6.27.18, driver version 2.0.44.14-NAPI
>
> Hi, e1000-devel
>
> Maybe I am not sending the quite proper email list to ask for
> an Intel 82599 driver issue. Thanks in advance if someone can give a
> reply!!
>
> Has someone ever tried to install an intel 82599(driver version
> 2.0.44.14-NAPI) under Linux 2.6.27.18?
>
> Intel 82599 driver's readme indicates that MultiQueue
> is always
> enabled, but we found it unable to be enabled even by "modprobe ixgbe
> MQ=1 RSS=1"
>
> Linux will claim the following msgs when the ixgbe driver inserted,
> firstly MQ enabled automatically, then RSS disabled and finally MQ
> disabled too caused by ixgbe_init_interrupt_scheme().
>
>
>
> ixgbe: Multiple Queue Support Enabled
>
> ixgbe: 0000:05:00.0: ixgbe_check_options: RSS is not supported on this
> hardware. Disabling RSS.
>
> .....
>
> ixgbe: 0000:05:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled:
> Rx Queue count = 1, Tx Queue count = 1
>
>
>
> Related logs attached. Are there any kernel/driver building options we
> missed at building time?
>
> Thanks a lot!
>
>
>
> Best Regards,
>
> Ocean
>
>
>
>
[Attachment #3 (--===============8709841539622506645==)]
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic