[prev in list] [next in list] [prev in thread] [next in thread]
List: xen-users
Subject: [Xen-users] FW: Loading PCIe Device Driver at Dom0
From: Kenneth Wong <kenwong () marvell ! com>
Date: 2012-05-23 23:49:11
Message-ID: F766E4F80769BD478052FB6533FA745D1A2FB28898 () SC-VEXCH4 ! marvell ! com
[Download RAW message or body]
Dear all,
I have a PCIe device driver that I have been using on various Linux distributions and \
Kernel versions (2.6.x - 3.x.y) successfully all along.
I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When \
I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 \
without any issue. I can do reads and write to the hardware device. But once booted \
to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 \
and the console just hangs.
From my debug messages, it appears it hangs because the driver doesn't receive any \
interrupt after a command is sent to the hardware device by writing a parameter to \
the mapped register. Once that register is written, the device is expected to DMA \
the command from the buffer allocated by the driver.
The things that I can only think of that might have caused the problem are 1) IRQ \
mapping issue, or 2) DMA mapping issue, which I am not sure.
What the driver does:
Set up a command buffer:
Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
unsigned long buf_addr = __pa(buf);
unsigned int buf_addr_low = (unsigned int)buf_addr;
Tell device about the buffer:
iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
Set up IRQ:
if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
(!pci_enable_msi(dev)))
{
if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, \
my_dev)) {
return -ENODEV;
}
my_dev->intr_mode = INTERRUPT_MSI;
}
Ask device to fetch command from buffer (Expect interrupt after this after device \
fetched the command from buf. But interrupt did not happen.): iowrite32(buf_offset, \
dev->pci_reg_map + FETCH_CMD_REG);
From dmesg, it looks like IRQ initialization is complete.
[ 241.743769] My_driver initialization
[ 241.743787] xen: registering gsi 16 triggering 0 polarity 1
[ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
[ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
[ 241.743801] Already setup the GSI :16
[ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64
/proc/interrupts:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 \
CPU7
......
......
339: 0 0 0 0 0 0 0 \
0 xen-pirq-msi my-driver
......
......
Any idea what might cause the problem?
Is there anything we have to be enable/disable, use different functions, or do \
differently in drivers written for Xen Dom0 environment regarding the following? 1) \
Allocating a DMA buffer in driver to allow the device to DMA stuffs. 2) \
Requesting MSI irq.
Please advise!
Thanks a lot in advance!!
Kenneth
[Attachment #3 (text/html)]
<html xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Arial;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:Arial;
color:navy;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1752506910;
mso-list-type:hybrid;
mso-list-template-ids:1367405010 67698705 67698713 67698715 67698703 67698713 \
67698715 67698703 67698713 67698715;} @list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Dear all,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I have a PCIe device driver that I have been using on
various Linux distributions and Kernel versions (2.6.x – 3.x.y)
successfully all along. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I recently set up a Xen environment with Linux Mint 12 and
Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via
insmod manually) successfully at Dom0 without any issue. I can do reads
and write to the hardware device. But once booted to Xen, the driver
failed to complete the driver load (via insmod manually) at Dom 0 and the
console just hangs.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>From my debug messages, it appears it hangs because the
driver doesn’t receive any interrupt after a command is sent to the
hardware device by writing a parameter to the mapped register. Once that
register is written, the device is expected to DMA the command from the buffer
allocated by the driver.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The things that I can only think of that might have caused
the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not
sure.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>What the driver does:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Set up a command buffer:<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>Buf_t *buf =
kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>unsigned long buf_addr = \
__pa(buf);<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>unsigned int buf_addr_low =
(unsigned int)buf_addr;<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Tell device about the buffer:<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>iowrite32(buf_addr_low,
dev->pci_reg_map + BUF_ADR__LOW);<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Set up IRQ:<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'> if \
(pci_find_capability(dev, PCI_CAP_ID_MSI) &&<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>
(!pci_enable_msi(dev)))<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'> \
{<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>
if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME,
my_dev))<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>
{<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>
return -ENODEV;<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>
}<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>
my_dev->intr_mode = INTERRUPT_MSI;<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'> \
}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Ask device to fetch command from buffer (Expect interrupt
after this after device fetched the command from buf. But interrupt did
not happen.):<o:p></o:p></span></font></p>
<p class=MsoNormal style='text-indent:.5in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>iowrite32(buf_offset,
dev->pci_reg_map + FETCH_CMD_REG);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>From dmesg, it looks like IRQ initialization is \
complete.<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743769] My_driver initialization <o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743787] xen: registering gsi 16 triggering 0 polarity \
1<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743793] xen_map_pirq_gsi: returning irq 16 for gsi \
16<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743801] Already setup the GSI :16<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=maroon
face=Arial><span style='font-size:10.0pt;font-family:Arial;color:maroon'>[
241.743815] my-driver 0000:02:00.0: setting latency timer to \
64<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>/proc/interrupts:<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=blue face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:blue'>
CPU0
CPU1
CPU2 CPU3
CPU4
CPU5
CPU6 CPU7<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=blue face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:blue'>……<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=blue face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:blue'>……<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=blue face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:blue'>339:
0
0
0
0
0
0
0 0
xen-pirq-msi \
my-driver<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=blue face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:blue'>……<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in'><font size=2 color=blue face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:blue'>……<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Any idea what might cause the \
problem?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Is there anything we have to be enable/disable, use
different functions, or do differently in drivers written for Xen Dom0
environment regarding the following?<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 \
lfo2'><![if !supportLists]><font size=2 face=Arial><span \
style='font-size:10.0pt;font-family:Arial'><span style='mso-list:Ignore'>1)<font \
size=1 face="Times New Roman"><span style='font:7.0pt "Times New \
Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
style='font-size:10.0pt;font-family:Arial'>Allocating a DMA buffer in driver to allow \
the device to DMA stuffs.<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 \
lfo2'><![if !supportLists]><font size=2 face=Arial><span \
style='font-size:10.0pt;font-family:Arial'><span style='mso-list:Ignore'>2)<font \
size=1 face="Times New Roman"><span style='font:7.0pt "Times New \
Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
style='font-size:10.0pt;font-family:Arial'>Requesting MSI \
irq.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Please advise!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks a lot in advance!!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Kenneth<o:p></o:p></span></font></p>
</div>
</body>
</html>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
--===============7562871868949512336==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic