[prev in list] [next in list] [prev in thread] [next in thread]
List: freebsd-hackers
Subject: Re: [EXTERNAL] pcib msix allocation in arm64
From: Andrew Turner <andrew () fubar ! geek ! nz>
Date: 2022-11-11 20:10:41
Message-ID: DD81C474-8EEA-4B24-A8EC-4B9AE5E9EFED () fubar ! geek ! nz
[Download RAW message or body]
> On 11 Nov 2022, at 19:07, Souradeep Chakrabarti <schakrabarti@microsoft.com> wrote:
>
>
>
>
> > -----Original Message-----
> > From: Souradeep Chakrabarti
> > Sent: Thursday, November 10, 2022 3:20 PM
> > To: Andrew Turner <andrew@fubar.geek.nz <mailto:andrew@fubar.geek.nz>>
> > Cc: Warner Losh <imp@bsdimp.com <mailto:imp@bsdimp.com>>; Wei Hu \
> > <weh@microsoft.com <mailto:weh@microsoft.com>>; freebsd- hackers@FreeBSD.org \
> > <mailto:hackers@FreeBSD.org>
> > Subject: RE: [EXTERNAL] pcib msix allocation in arm64
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Andrew Turner <andrew@fubar.geek.nz>
> > > Sent: Thursday, November 3, 2022 6:21 PM
> > > To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> > > Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> > freebsd-
> > > hackers@FreeBSD.org
> > > Subject: Re: [EXTERNAL] pcib msix allocation in arm64
> > >
> > > Hi Souradeep,
> > >
> > > For the vmbus_pcib driver you'll need to implement the pcib_alloc_msi,
> > > pcib_release_msi, and the msix versions, and pcib_map_msi.
> > >
> > > For alloc and release you can just call into the same intr_* function
> > > as pci_host_generic_acpi.c. You'll need to pass in the xref for the
> > > interrupt controller. It needs to be the xref the MSI controller
> > > registered. For the GICv2m it's ACPI_MSI_XREF.
> > >
> > [Souradeep]
> > I have used following in vmbus_pcib.c :
> > ret = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is failing in
> > pic_lookup() with ESRCH by not finding pic with ACPI_MSI_XREF.
> > Do I need to do anything for pic_lookup() to be successful?
> >
> [Souradeep]
> I found out the reason here pic_lookup() for ACPI_MSI_XREF is failing, as
> we are not calling intr_msi_register() but intr_pic_register() is getting called \
> for the PCIB. This is because we don't have ITS implemented in Hyper-V. Instead, \
> Hyper-V is presenting the MSI/MSI-X as SPI because of it's own limitation.
> Linux has solved it using custom irqchip driver to handle it.
> But I am not sure how to address it in FreeBSD.
> Can you please point me out something here?
We currently only support SPIs in the GICv3 driver under FDT. To teach the ACPI \
attachment to support them you'll need to set sc->gic_mbi_start and sc->gic_mbi_end \
in sys/arm64/arm64/gic_v3_acpi.c to the correct value from the ACPI tables before \
calling gic_v3_attach. After this returns successfully you can then call \
intr_msi_register.
See gic_v3_fdt.c for how we do it under FDT.
Andrew
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote \
type="cite" class=""><div class="">On 11 Nov 2022, at 19:07, Souradeep Chakrabarti \
<<a href="mailto:schakrabarti@microsoft.com" \
class="">schakrabarti@microsoft.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: \
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; \
font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote \
type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; \
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; \
text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">-----Original \
Message-----<br class="">From: Souradeep Chakrabarti<br class="">Sent: Thursday, \
November 10, 2022 3:20 PM<br class="">To: Andrew Turner <<a \
href="mailto:andrew@fubar.geek.nz" class="">andrew@fubar.geek.nz</a>><br \
class="">Cc: Warner Losh <<a href="mailto:imp@bsdimp.com" \
class="">imp@bsdimp.com</a>>; Wei Hu <<a href="mailto:weh@microsoft.com" \
class="">weh@microsoft.com</a>>; freebsd-<br class=""><a \
href="mailto:hackers@FreeBSD.org" class="">hackers@FreeBSD.org</a><br \
class="">Subject: RE: [EXTERNAL] pcib msix allocation in arm64<br class=""><br \
class=""><br class=""><br class=""><br class=""><blockquote type="cite" \
class="">-----Original Message-----<br class="">From: Andrew Turner <<a \
href="mailto:andrew@fubar.geek.nz" class="">andrew@fubar.geek.nz</a>><br \
class="">Sent: Thursday, November 3, 2022 6:21 PM<br class="">To: Souradeep \
Chakrabarti <<a href="mailto:schakrabarti@microsoft.com" \
class="">schakrabarti@microsoft.com</a>><br class="">Cc: Warner Losh <<a \
href="mailto:imp@bsdimp.com" class="">imp@bsdimp.com</a>>; Wei Hu <<a \
href="mailto:weh@microsoft.com" class="">weh@microsoft.com</a>>;<br \
class=""></blockquote>freebsd-<br class=""><blockquote type="cite" class=""><a \
href="mailto:hackers@FreeBSD.org" class="">hackers@FreeBSD.org</a><br \
class="">Subject: Re: [EXTERNAL] pcib msix allocation in arm64<br class=""><br \
class="">Hi Souradeep,<br class=""><br class="">For the vmbus_pcib driver you'll need \
to implement the pcib_alloc_msi,<br class="">pcib_release_msi, and the msix versions, \
and pcib_map_msi.<br class=""><br class="">For alloc and release you can just call \
into the same intr_* function<br class="">as pci_host_generic_acpi.c. You'll need to \
pass in the xref for the<br class="">interrupt controller. It needs to be the xref \
the MSI controller<br class="">registered. For the GICv2m it's ACPI_MSI_XREF.<br \
class=""><br class=""></blockquote>[Souradeep]<br class="">I have used following in \
vmbus_pcib.c :<br class="">ret = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But \
it is failing in<br class="">pic_lookup() with ESRCH by not finding pic with \
ACPI_MSI_XREF.<br class="">Do I need to do anything for pic_lookup() to be \
successful?<br class=""><br class=""></blockquote><span style="caret-color: rgb(0, 0, \
0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline \
!important;" class="">[Souradeep]<span \
class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, \
0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">I found out the reason here \
pic_lookup() for ACPI_MSI_XREF is failing, as</span><br style="caret-color: rgb(0, 0, \
0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">we are not calling \
intr_msi_register() but intr_pic_register() is getting called for the PCIB.</span><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" class="">This is \
because we don't have ITS implemented in Hyper-V. Instead, Hyper-V is presenting \
the</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: \
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; \
font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline \
!important;" class="">MSI/MSI-X as SPI because of it's own limitation.</span><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" class="">Linux has \
solved it using custom irqchip driver to handle it.<span \
class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, \
0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">But I am not sure how to address \
it in FreeBSD.<span class="Apple-converted-space"> </span></span><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" class="">Can you \
please point me out something here?</span><br style="caret-color: rgb(0, 0, 0); \
font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""></div></blockquote><div><br class=""></div><div>We currently only support \
SPIs in the GICv3 driver under FDT. To teach the ACPI attachment to support them \
you'll need to set sc->gic_mbi_start and sc->gic_mbi_end in \
sys/arm64/arm64/gic_v3_acpi.c to the correct value from the ACPI tables before \
calling gic_v3_attach. After this returns successfully you can then \
call intr_msi_register.</div><div><br class=""></div><div>See gic_v3_fdt.c \
for how we do it under FDT.</div><div><br \
class=""></div><div>Andrew</div></div></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic