[prev in list] [next in list] [prev in thread] [next in thread]
List: gnuradio-discuss
Subject: Re: [announcement] master: SWIG -> Pybind11 (breaks OOT interface)
From: "Morman, Joshua" <jmorman () perspectalabs ! com>
Date: 2020-06-26 16:37:28
Message-ID: BN3P110MB049807690C542716CFD623A7A6930 () BN3P110MB0498 ! NAMP110 ! PROD ! OUTLOOK ! COM
[Download RAW message or body]
Thank you Marcus for the merge, the introduction, and the warnings :)
For all the Ubuntu users - the GNU Radio Master PPA has been updated with builds \
pulled from the latest master, so it now includes the pybind11 bindings, and OOTs \
will need to start migrating to the new python binding structure. Only install from \
this PPA if you want to test the latest master branch and get your OOTs ready for 3.9 \
(https://wiki.gnuradio.org/index.php/GNU_Radio_3.9_OOT_Module_Porting_Guide). \
Otherwise, it is better to stay with the gnuradio-releases PPA. Here is the PPA \
info for master:
sudo add-apt-repository ppa:gnuradio/gnuradio-master
sudo apt-get update
sudo apt-get install gnuradio
Fedora COPR is not yet up to date, but I will let everyone know when that supports \
the current master branch with pybind11.
Josh
--------------------------------------------------------------------------------
From: Marcus Müller <mmueller@gnuradio.org<mailto:mmueller@gnuradio.org>>
Date: Thu, Jun 18, 2020 at 2:59 PM
Subject: [announcement] master: SWIG -> Pybind11 (breaks OOT interface)
To: Gnuradio Mailing List <discuss-gnuradio@gnu.org<mailto:discuss-gnuradio@gnu.org>>
Hello, most prolific SDR community to wander this wide globe,
TL;DR: brace yourself. If you're on master, we'll be breaking your OOT.
You don't have to worry about anything if you're using GNU Radio 3.8.
We're close to merging the pybind branch into master – and that means
we're improving things a lot.
Formerly, we used SWIG to generate Python bindings for GNU Radio's C++
classes – a decision made much more than a decade ago[1], when it still
wasn't clear that Python would be the only language we support (SWIG can
generate wrappers for *a lot* of languages).
In what it does, SWIG is pretty impressive: give it some C++ header, and
it will generate the correct binding for python, including generating of
some mapper types, transparent handling of pointers etc. And, we've
really been using it a lot and in a lot of complex usage scenarios. I
honestly think that, together with OpenCV and LLDB, we really might be
the healthiest and largest project to use SWIG in such a wide way. I'm
very thankful SWIG exists!
Now, having said that, SWIG was always kind of a pain to work into our
source tree. The promise of "throwing in unmodified C++ headers, getting
out perfect Python bindings", far too often, turned out to be "after
massaging these headers quite a bit, then adding a bit of SWIG
directives here and there in swig .i files that we have to keep updated
whenever we change header structure"; and every time something broke our
SWIG bindings, be it a bug on our side or an unexpected change in SWIG
semantics, or be it that SWIG stubbornly refuses to use the include
paths we specify on some platforms, we sunk literal working days into
getting things fixed.
So, as awesome as SWIG's automatism are, using it on such a type-diverse
code base, with such a platform-diverse user community, was kind of
liable for a lot of time not spent on developing a better SDR framework,
but fixing tooling.
Thus, Josh took it upon himself to get this sorted out – and replace
SWIG with Pybind11. Now, Pybind11 is pretty different from SWIG. It's
actually pure C++11 code, and just compiles with standard compilers. No
extra tooling required.
It does come at the expense of you having to declare your interface
yourself. That sounds like a regression, but honestly, there's been so
much work put into making this automateable by Josh, for most blocks
that don't do anything "special", you can just let the automatism take
over. If that doesn't work, you'll typically get a compiler error,
telling you exactly what's wrong. To illustrate, I'll attach the code
generated to wrap a simple block below[2].
However, what that means is that we're also changing the way OOTs need
to wrap their C++ to Python so that they can still can work with GNU
Radio C++ objects wrapped to Python. gr_modtool has been updated to have
new stubs, and the aforementioned automatism is available, too. Small
problem: unlike compilation, the code generation does need an extra bit
of tooling, pygccxml to be specific. This is really only necessary when
you wrap a block that's not been wrapped before, not when compiling any
existing module or the GNU Radio source tree.
In terms of compile time: according to my current level of overview, for
proper multicore machines, building the many smaller C++ files tends to
be a bit faster than first running SWIG to generate 100kLOC-files and
then compile these monsters. For single- or dualcore machines, the
opposite tends to be true. It's my duty to remind you that compiling on
a RaspberryPi itself might get even more cumbersome than it currently
is, and that you should prefer the cross-building toolchains of the
operating system you're running on the embedded platform rather than
building directly on it. Especially debian makes that pretty easy.
Best regards,
and the happiest of hacking,
Marcus
[1]
https://lists.gnu.org/archive/html/discuss-gnuradio/2003-03/msg00135.html
[2]
py::class_<pub_sink,
gr::sync_block,
gr::block,
gr::basic_block,
std::shared_ptr<pub_sink>>(m, "pub_sink", D(pub_sink))
.def(py::init(&pub_sink::make),
py::arg("itemsize"),
py::arg("vlen"),
py::arg("address"),
py::arg("timeout") = 100,
py::arg("pass_tags") = false,
py::arg("hwm") = -1,
py::arg("key") = "",
D(pub_sink, make))
.def("last_endpoint", &pub_sink::last_endpoint, D(pub_sink, last_endpoint))
;
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} \
</style> </head>
<body dir="ltr">
<div>Thank you Marcus for the merge, the introduction, and the warnings :)<br>
</div>
<div><br>
</div>
<div>For all the Ubuntu users - the GNU Radio Master PPA has been updated with builds \
pulled from the latest master, so it now includes the pybind11 bindings, and OOTs \
will need to start migrating to the new python binding structure. Only install \
from this PPA if you want to test the latest master branch and get your OOTs ready \
for 3.9 (<a href="https://wiki.gnuradio.org/index.php/GNU_Radio_3.9_OOT_Module_Porting_Guide" \
id="LPlnk466830">https://wiki.gnuradio.org/index.php/GNU_Radio_3.9_OOT_Module_Porting_Guide</a>).
Otherwise, it is better to stay with the gnuradio-releases PPA. Here is \
the PPA info for master:<br> </div>
<div><br>
</div>
<div>
<pre>sudo add-apt-repository ppa:gnuradio/gnuradio-master
sudo apt-get update<br>sudo apt-get install gnuradio<br></pre>
</div>
<div><br>
</div>
<div>Fedora COPR is not yet up to date, but I will let everyone know when that \
supports the current master branch with pybind11.<br> </div>
<div><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0)"> Josh<br>
</div>
<div><br>
<div>
<div dir="ltr">--------------------------------------------------------------------------------<br>
</div>
<div dir="ltr"><br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">From: <b class="x_gmail_sendername" dir="auto">
Marcus Müller</b> <span dir="auto"><<a \
href="mailto:mmueller@gnuradio.org">mmueller@gnuradio.org</a>></span><br>
Date: Thu, Jun 18, 2020 at 2:59 PM<br>
Subject: [announcement] master: SWIG -> Pybind11 (breaks OOT interface)<br>
To: Gnuradio Mailing List <<a \
href="mailto:discuss-gnuradio@gnu.org">discuss-gnuradio@gnu.org</a>><br> </div>
<br>
<br>
Hello, most prolific SDR community to wander this wide globe,<br>
<br>
TL;DR: brace yourself. If you're on master, we'll be breaking your OOT. <br>
You don't have to worry about anything if you're using GNU Radio 3.8.<br>
<br>
We're close to merging the pybind branch into master – and that means <br>
we're improving things a lot.<br>
<br>
Formerly, we used SWIG to generate Python bindings for GNU Radio's C++ <br>
classes – a decision made much more than a decade ago[1], when it still <br>
wasn't clear that Python would be the only language we support (SWIG can <br>
generate wrappers for *a lot* of languages).<br>
<br>
In what it does, SWIG is pretty impressive: give it some C++ header, and <br>
it will generate the correct binding for python, including generating of <br>
some mapper types, transparent handling of pointers etc. And, we've <br>
really been using it a lot and in a lot of complex usage scenarios. I <br>
honestly think that, together with OpenCV and LLDB, we really might be <br>
the healthiest and largest project to use SWIG in such a wide way. I'm <br>
very thankful SWIG exists!<br>
<br>
Now, having said that, SWIG was always kind of a pain to work into our <br>
source tree. The promise of "throwing in unmodified C++ headers, getting \
<br> out perfect Python bindings", far too often, turned out to be "after \
<br> massaging these headers quite a bit, then adding a bit of SWIG <br>
directives here and there in swig .i files that we have to keep updated <br>
whenever we change header structure"; and every time something broke our <br>
SWIG bindings, be it a bug on our side or an unexpected change in SWIG <br>
semantics, or be it that SWIG stubbornly refuses to use the include <br>
paths we specify on some platforms, we sunk literal working days into <br>
getting things fixed.<br>
<br>
So, as awesome as SWIG's automatism are, using it on such a type-diverse <br>
code base, with such a platform-diverse user community, was kind of <br>
liable for a lot of time not spent on developing a better SDR framework, <br>
but fixing tooling.<br>
<br>
Thus, Josh took it upon himself to get this sorted out – and replace <br>
SWIG with Pybind11. Now, Pybind11 is pretty different from SWIG. It's <br>
actually pure C++11 code, and just compiles with standard compilers. No <br>
extra tooling required.<br>
<br>
It does come at the expense of you having to declare your interface <br>
yourself. That sounds like a regression, but honestly, there's been so <br>
much work put into making this automateable by Josh, for most blocks <br>
that don't do anything "special", you can just let the automatism take <br>
over. If that doesn't work, you'll typically get a compiler error, <br>
telling you exactly what's wrong. To illustrate, I'll attach the code <br>
generated to wrap a simple block below[2].<br>
<br>
However, what that means is that we're also changing the way OOTs need <br>
to wrap their C++ to Python so that they can still can work with GNU <br>
Radio C++ objects wrapped to Python. gr_modtool has been updated to have <br>
new stubs, and the aforementioned automatism is available, too. Small <br>
problem: unlike compilation, the code generation does need an extra bit <br>
of tooling, pygccxml to be specific. This is really only necessary when <br>
you wrap a block that's not been wrapped before, not when compiling any <br>
existing module or the GNU Radio source tree.<br>
<br>
In terms of compile time: according to my current level of overview, for <br>
proper multicore machines, building the many smaller C++ files tends to <br>
be a bit faster than first running SWIG to generate 100kLOC-files and <br>
then compile these monsters. For single- or dualcore machines, the <br>
opposite tends to be true. It's my duty to remind you that compiling on <br>
a RaspberryPi itself might get even more cumbersome than it currently <br>
is, and that you should prefer the cross-building toolchains of the <br>
operating system you're running on the embedded platform rather than <br>
building directly on it. Especially debian makes that pretty easy.<br>
<br>
Best regards,<br>
and the happiest of hacking,<br>
<br>
Marcus<br>
<br>
[1] <br>
<a href="https://lists.gnu.org/archive/html/discuss-gnuradio/2003-03/msg00135.html" \
rel="noreferrer" target="_blank">https://lists.gnu.org/archive/html/discuss-gnuradio/2003-03/msg00135.html</a><br>
<br>
[2]<br>
<br>
py::class_<pub_sink,<br>
gr::sync_block,<br>
gr::block,<br>
gr::basic_block,<br>
std::shared_ptr<pub_sink>>(m, \
"pub_sink", D(pub_sink))<br>
.def(py::init(&pub_sink::make),<br>
py::arg("itemsize"),<br>
py::arg("vlen"),<br>
py::arg("address"),<br>
py::arg("timeout") = 100,<br>
py::arg("pass_tags") = false,<br>
py::arg("hwm") = -1,<br>
py::arg("key") = "",<br>
D(pub_sink, make))<br>
.def("last_endpoint", &pub_sink::last_endpoint, D(pub_sink, \
last_endpoint))<br> ;<br>
<br>
<br>
</div>
</div>
</div>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic