[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-bindings
Subject: Re: [Kde-bindings] Progress report on Py*5 binding generation
From: Shaheed Haque <srhaque () theiet ! org>
Date: 2016-04-21 23:29:26
Message-ID: CAHAc2jdPOKqpW71C6hYLcK=Dcwv6UPQmgHW0Cyh5AHjP5tSrBQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Steve,
On 21 April 2016 at 23:29, Stephen Kelly <steveire@gmail.com> wrote:
> Shaheed Haque wrote:
>
> >> To that end, in addition to sip/kselectionproxymodel.sip I now have:
> >>
> >> $ cat sip/kitemmodelsmod.sip
> >>
> >> %Module PyKDE5.kitemmodels
> >>
> >> %ModuleHeaderCode
> >> #pragma GCC visibility push(default)
> >> %End
> >>
> >> %Import QtCore/QtCoremod.sip
> >>
> >> %Include kselectionproxymodel.sip
> >>
> >>
> >> Is this what I am supposed to do? Here is the output I get when
> >> attempting to use it:
> >
> > Yeeessss...it is a bit tricky...
> >
> > First, if you use the "-v" flag to the sip_compiler.py, it will show you
> > the exact command like for SIP and later the C++ compiler/linker. That
> > might help you see what is going on.
>
> Yes, that's good, thanks.
>
> > The %Feature thing you may be able to ignore. It is used along with the
> > SIP compiler's -x (lower case X) option to try to work around the problem
> > where in KF5, you often get AAAmod.sip and BBBmod.sip which need to
> import
> > each other
>
> You have alluded to this several times, but you have never stated which
> modules form a cycle like this. Which KF5 libraries have this cycle?
>
I think there are many of these. One example is this one:
sip/KIOCore/KIOCoremod.sip:%Import KIOCore/kio/kiomod.sip
sip/KIOCore/kio/kiomod.sip:%Import KIOCore/KIOCoremod.sip
This was the first one I happened on, and there are others. I have not
tried to track down the exact root cause but I guess that this does not
often show up in normal C++ development because of the effect of header
guard macros.
> > ...I used the SIP compiler's -X (upper case X) option to extract the
> > %Extract data, and generate the "-I foo -I bar" paths for the C++
> > compiler. So, you can just put a %Extract section like this into your
> > file:
> >
> > %Extract(id=includes)
> > /usr/include/KF5/KDBusAddons
> > /usr/include/x86_64-linux-gnu/qt5/QtCore
> > /usr/include/x86_64-linux-gnu/qt5/QtDBus
> > %End
>
> As the file might be generated by cmake, this might work, but it seems
> easier to pass the include directories on the command line. The includes
> will be different on each platform.
>
That is correct, but AFAIK, this is just a build time dependency. Once the
modules are built, nobody will care. However, I'm sensitive that my
ignorance about how CMake is used by packagers might make the current
approach suboptimal.
> After hacking the sip_compiler a bit I can do this:
>
> ./sip_compiler.py -v --includes "/usr/include/x86_64-linux-
> gnu/qt5,/usr/include/x86_64-linux-gnu/qt5/QtCore" --select
> @kitemmodelsmod.sip sip cxx
>
>
> My hacking is quite dirty, but you can extract feature requests from it,
> namely, don't require the include/extract stuff.
>
For me, and for now at least, it is important to be able to run on a large
body of code without generating anything by hand so generating xxxmod.sip
files by hand (with or without the %Extract) is not practical. Perhaps a
CLI option would be acceptable?
> The makefile manipulation stuff can be ignored, because the SIP generated
> makefiles won't play nicely with cmake, but should be easily able to
> generate our own makefiles/ninja build files.
>
OK. Once we get there, then maybe some of the stuff above might be worth
reconsidering too.
> diff --git a/sip_generation/sip_compiler.py
> b/sip_generation/sip_compiler.py
> index 057dfe9..715f9c4 100755
> --- a/sip_generation/sip_compiler.py
> +++ b/sip_generation/sip_compiler.py
> @@ -133,20 +133,23 @@ class CxxDriver(object):
> if line.startswith("%Module"):
> feature_list = os.path.join(self.input_dir,
> "modules.features")
> tmp = set()
> - with open(feature_list, "rU") as f:
> - for feature in f:
> - if feature not in tmp:
> - tmp.add(feature)
> - o.write(feature)
> + # with open(feature_list, "rU") as f:
> + # for feature in f:
> + # if feature not in tmp:
> + # tmp.add(feature)
> + # o.write(feature)
> logger.debug(modified_source)
> feature = sip_file.replace(os.path.sep, "_").replace(".", "_")
> - cmd = [self.sipconfig.sip_bin, "-c", full_output, "-b",
> build_file, "-x", feature, "-X",
> - INCLUDES_EXTRACT + ":" + module_includes] +
> self.pyqt_sip_flags + sip_roots + [modified_source]
> + cmd = [self.sipconfig.sip_bin, "-c", full_output, "-b",
> build_file, "-x", feature] + self.pyqt_sip_flags + sip_roots +
> [modified_source]
> self._run_command(cmd)
> #
> # Create the Makefile.
> #
> - module_includes = self.includes + open(module_includes,
> "rU").read().split("\n")
> + module_includes = self.includes
> + try:
> + module_includes += open(module_includes,
> "rU").read().split("\n")
> + except:
> + pass
> self.sipconfig._macros["INCDIR"] = " ".join(module_includes)
> makefile = sipconfig.SIPModuleMakefile(self.sipconfig,
> build_file, makefile=make_file)
> #
> @@ -155,6 +158,9 @@ class CxxDriver(object):
> # ".dll" extension on Windows).
> #makefile.extra_libs = ["KParts"]
> #
> + makefile.extra_cxxflags = ["-std=c++14"]
> + makefile.extra_lflags = ["-std=c++14", "-Wl,--fatal-warnings",
> "-Wl,--no-undefined"]
> + makefile.extra_libs = ["m /usr/lib/x86_64-linux-
> gnu/libQt5Core.so /usr/lib/x86_64-linux-gnu/libKF5ItemModels.so
> /usr/lib/x86_64-linux-gnu/libpython2.7.so"]
> makefile.generate()
> self._run_command(["make", "-f", os.path.basename(make_file)],
> cwd=full_output)
> except Exception as e
>
>
> With that done here is my output:
>
> $ ./sip_compiler.py -v --includes "/usr/include/x86_64-linux-
> gnu/qt5,/usr/include/x86_64-linux-gnu/qt5/QtCore" --select
> @kitemmodelsmod.sip sip cxx
> 2016-04-22 00:23:01,282 __main__ INFO: Creating cxx/
> 2016-04-22 00:23:01,283 __main__ DEBUG: cxx/kitemmodelsmod.sip.tmp
> 2016-04-22 00:23:01,283 __main__ INFO: /usr/bin/sip -c cxx/ -b
> cxx/module.sbf -x kitemmodelsmod_sip -x VendorID -t WS_X11 -t Qt_5_4_2 -x
> Py_v3 -I/usr/share/sip/PyQt5 -Isip cxx/kitemmodelsmod.sip.tmp
> 2016-04-22 00:23:01,728 __main__ INFO: make -f module.Makefile
> g++ -c -std=c++14 -g -O2 -fstack-protector-strong -Wformat -Werror=format-
> security -D_FORTIFY_SOURCE=2 -fPIC -Wall -W -DNDEBUG -I. -
> I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-
> gnu/qt5/QtCore -I/usr/include/python2.7 -o sipkitemmodelscmodule.o
> sipkitemmodelscmodule.cpp
> g++ -c -std=c++14 -g -O2 -fstack-protector-strong -Wformat -Werror=format-
> security -D_FORTIFY_SOURCE=2 -fPIC -Wall -W -DNDEBUG -I. -
> I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-
> gnu/qt5/QtCore -I/usr/include/python2.7 -o
> sipkitemmodelsKSelectionProxyModel.o sipkitemmodelsKSelectionProxyModel.cpp
> g++ -std=c++14 -Wl,--fatal-warnings -Wl,--no-undefined -Wl,-Bsymbolic-
> functions -Wl,-z,relro -shared -Wl,--version-script=kitemmodels.exp -o
> kitemmodels.so sipkitemmodelscmodule.o sipkitemmodelsKSelectionProxyModel.o
> -lm /usr/lib/x86_64-linux-gnu/libQt5Core.so /usr/lib/x86_64-linux-
> gnu/libKF5ItemModels.so /usr/lib/x86_64-linux-gnu/libpython2.7.so
>
> That created the cxx/kitemmodels.so library
>
> $ ls cxx/
> kitemmodels.exp sipAPIkitemmodels.h
> kitemmodelsmod.sip.tmp sipkitemmodelscmodule.cpp
> kitemmodels.so* sipkitemmodelscmodule.o
> module.Makefile sipkitemmodelsKSelectionProxyModel.cpp
> module.sbf sipkitemmodelsKSelectionProxyModel.o
> pythontest.py*
>
> $ cat cxx/pythontest.py
> #!/usr/bin/env python
>
> from PyQt5 import QtCore
>
> import kitemmodels
>
> $ ./cxx/pythontest.py
> Traceback (most recent call last):
> File "./cxx/pythontest.py", line 5, in <module>
> import kitemmodels
> SystemError: dynamic module not initialized properly
>
>
> Is there some other step to take before that import will work?
>
Only to pick up the code I pushed earlier tonight [1]. This brings a slight
change the in the CLI for the sip_compiler.py. The second argument ("cxx"
above) is now optional and defaulted from the project rules "project_name".
For KF5, the latter is PyKF5, and the net effect of the changes is that you
get the following by default:
- a directory PyKF5
- containing a __init__.py
- and all the .so files such as KItemModels.so
- and a disposable tmp directory
so that you can then just say "from PyKF5 import KItemModels". You'll
notice that in support of this, there is a temporary list of hardcoded
libraries to link against (a quick hack just to get us going).
Thanks, Shaheed
[1] See
https://www.riverbankcomputing.com/pipermail/pyqt/2016-April/037331.html
for the details.
> Thanks,
>
> Steve.
> _______________________________________________
> Kde-bindings mailing list
> Kde-bindings@kde.org
> https://mail.kde.org/mailman/listinfo/kde-bindings
>
[Attachment #5 (text/html)]
<div dir="ltr">Hi Steve,<br><div class="gmail_extra"><br><div class="gmail_quote">On \
21 April 2016 at 23:29, Stephen Kelly <span dir="ltr"><<a \
href="mailto:steveire@gmail.com" target="_blank">steveire@gmail.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>Shaheed Haque \
wrote:<br> <br>
>> To that end, in addition to sip/kselectionproxymodel.sip I now have:<br>
>><br>
>> $ cat sip/kitemmodelsmod.sip<br>
>><br>
>> %Module PyKDE5.kitemmodels<br>
>><br>
>> %ModuleHeaderCode<br>
>> #pragma GCC visibility push(default)<br>
>> %End<br>
>><br>
>> %Import QtCore/QtCoremod.sip<br>
>><br>
>> %Include kselectionproxymodel.sip<br>
>><br>
>><br>
>> Is this what I am supposed to do? Here is the output I get when<br>
>> attempting to use it:<br>
><br>
> Yeeessss...it is a bit tricky...<br>
><br>
> First, if you use the "-v" flag to the sip_compiler.py, it will show \
you<br> > the exact command like for SIP and later the C++ compiler/linker. \
That<br> > might help you see what is going on.<br>
<br>
</span>Yes, that's good, thanks.<br>
<span><br>
> The %Feature thing you may be able to ignore. It is used along with the<br>
> SIP compiler's -x (lower case X) option to try to work around the \
problem<br> > where in KF5, you often get AAAmod.sip and BBBmod.sip which need to \
import<br> > each other<br>
<br>
</span>You have alluded to this several times, but you have never stated which<br>
modules form a cycle like this. Which KF5 libraries have this \
cycle?<span><br></span></blockquote><div><br></div><div>I think there are many of \
these. One example is this one:<br><br>sip/KIOCore/KIOCoremod.sip:%Import \
KIOCore/kio/kiomod.sip<br>sip/KIOCore/kio/kiomod.sip:%Import \
KIOCore/KIOCoremod.sip<br><br></div><div>This was the first one I happened on, and \
there are others. I have not tried to track down the exact root cause but I guess \
that this does not often show up in normal C++ development because of the effect of \
header guard macros.<br></div><div> </div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><span> > ...I used the SIP compiler's -X \
(upper case X) option to extract the<br> > %Extract data, and generate the \
"-I foo -I bar" paths for the C++<br> > compiler. So, you can just put a \
%Extract section like this into your<br> > file:<br>
><br>
> %Extract(id=includes)<br>
> /usr/include/KF5/KDBusAddons<br>
> /usr/include/x86_64-linux-gnu/qt5/QtCore<br>
> /usr/include/x86_64-linux-gnu/qt5/QtDBus<br>
> %End<br>
<br>
</span>As the file might be generated by cmake, this might work, but it seems<br>
easier to pass the include directories on the command line. The includes<br>
will be different on each platform.<br></blockquote><div><br></div><div>That is \
correct, but AFAIK, this is just a build time dependency. Once the modules are built, \
nobody will care. However, I'm sensitive that my ignorance about how CMake is \
used by packagers might make the current approach suboptimal.<br></div><div> \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"> After hacking the sip_compiler a bit I can \
do this:<br> <br>
./sip_compiler.py -v --includes "/usr/include/x86_64-linux-<br>
gnu/qt5,/usr/include/x86_64-linux-gnu/qt5/QtCore" --select<br>
@kitemmodelsmod.sip sip cxx<br>
<br>
<br>
My hacking is quite dirty, but you can extract feature requests from it,<br>
namely, don't require the include/extract \
stuff.<br></blockquote><div><br></div><div>For me, and for now at least, it is \
important to be able to run on a large body of code without generating anything by \
hand so generating xxxmod.sip files by hand (with or without the %Extract) is not \
practical. Perhaps a CLI option would be acceptable?<br></div><div> \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"> The makefile manipulation stuff can be \
ignored, because the SIP generated<br> makefiles won't play nicely with cmake, \
but should be easily able to<br> generate our own makefiles/ninja build \
files.<br></blockquote><div><br></div><div>OK. Once we get there, then maybe some of \
the stuff above might be worth reconsidering too.<br> <br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">
diff --git a/sip_generation/sip_compiler.py b/sip_generation/sip_compiler.py<br>
index 057dfe9..715f9c4 100755<br>
--- a/sip_generation/sip_compiler.py<br>
+++ b/sip_generation/sip_compiler.py<br>
@@ -133,20 +133,23 @@ class CxxDriver(object):<br>
if line.startswith("%Module"):<br>
feature_list = \
os.path.join(self.input_dir,<br> "modules.features")<br>
tmp = set()<br>
- with open(feature_list, "rU") as \
f:<br>
- for feature in f:<br>
- if feature not in tmp:<br>
- tmp.add(feature)<br>
- o.write(feature)<br>
+ # with open(feature_list, "rU") \
as f:<br> + # for feature in f:<br>
+ # if feature not in tmp:<br>
+ # tmp.add(feature)<br>
+ # o.write(feature)<br>
logger.debug(modified_source)<br>
feature = sip_file.replace(os.path.sep, \
"_").replace(".", "_")<br>
- cmd = [self.sipconfig.sip_bin, "-c", full_output, \
"-b",<br> build_file, "-x", feature, "-X",<br>
- INCLUDES_EXTRACT + ":" + module_includes] \
+<br> self.pyqt_sip_flags + sip_roots + [modified_source]<br>
+ cmd = [self.sipconfig.sip_bin, "-c", full_output, \
"-b",<br> build_file, "-x", feature] + self.pyqt_sip_flags + \
sip_roots +<br> [modified_source]<br>
self._run_command(cmd)<br>
#<br>
# Create the Makefile.<br>
#<br>
- module_includes = self.includes + open(module_includes,<br>
"rU").read().split("\n")<br>
+ module_includes = self.includes<br>
+ try:<br>
+ module_includes += open(module_includes,<br>
"rU").read().split("\n")<br>
+ except:<br>
+ pass<br>
self.sipconfig._macros["INCDIR"] = " \
".join(module_includes)<br>
makefile = sipconfig.SIPModuleMakefile(self.sipconfig,<br>
build_file, makefile=make_file)<br>
#<br>
@@ -155,6 +158,9 @@ class CxxDriver(object):<br>
# ".dll" extension on Windows).<br>
#makefile.extra_libs = ["KParts"]<br>
#<br>
+ makefile.extra_cxxflags = ["-std=c++14"]<br>
+ makefile.extra_lflags = ["-std=c++14", \
"-Wl,--fatal-warnings",<br> "-Wl,--no-undefined"]<br>
+ makefile.extra_libs = ["m /usr/lib/x86_64-linux-<br>
gnu/libQt5Core.so /usr/lib/x86_64-linux-gnu/libKF5ItemModels.so<br>
/usr/lib/x86_64-linux-gnu/<a href="http://libpython2.7.so" rel="noreferrer" \
target="_blank">libpython2.7.so</a>"]<br> makefile.generate()<br>
self._run_command(["make", "-f", \
os.path.basename(make_file)],<br> cwd=full_output)<br>
except Exception as e<br>
<br>
<br>
With that done here is my output:<br>
<br>
$ ./sip_compiler.py -v --includes "/usr/include/x86_64-linux-<br>
gnu/qt5,/usr/include/x86_64-linux-gnu/qt5/QtCore" --select<br>
@kitemmodelsmod.sip sip cxx<br>
2016-04-22 00:23:01,282 __main__ INFO: Creating cxx/<br>
2016-04-22 00:23:01,283 __main__ DEBUG: cxx/kitemmodelsmod.sip.tmp<br>
2016-04-22 00:23:01,283 __main__ INFO: /usr/bin/sip -c cxx/ -b<br>
cxx/module.sbf -x kitemmodelsmod_sip -x VendorID -t WS_X11 -t Qt_5_4_2 -x<br>
Py_v3 -I/usr/share/sip/PyQt5 -Isip cxx/kitemmodelsmod.sip.tmp<br>
2016-04-22 00:23:01,728 __main__ INFO: make -f module.Makefile<br>
g++ -c -std=c++14 -g -O2 -fstack-protector-strong -Wformat -Werror=format-<br>
security -D_FORTIFY_SOURCE=2 -fPIC -Wall -W -DNDEBUG -I. -<br>
I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-<br>
gnu/qt5/QtCore -I/usr/include/python2.7 -o sipkitemmodelscmodule.o<br>
sipkitemmodelscmodule.cpp<br>
g++ -c -std=c++14 -g -O2 -fstack-protector-strong -Wformat -Werror=format-<br>
security -D_FORTIFY_SOURCE=2 -fPIC -Wall -W -DNDEBUG -I. -<br>
I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-<br>
gnu/qt5/QtCore -I/usr/include/python2.7 -o<br>
sipkitemmodelsKSelectionProxyModel.o sipkitemmodelsKSelectionProxyModel.cpp<br>
g++ -std=c++14 -Wl,--fatal-warnings -Wl,--no-undefined -Wl,-Bsymbolic-<br>
functions -Wl,-z,relro -shared -Wl,--version-script=kitemmodels.exp -o<br>
kitemmodels.so sipkitemmodelscmodule.o sipkitemmodelsKSelectionProxyModel.o<br>
-lm /usr/lib/x86_64-linux-gnu/libQt5Core.so /usr/lib/x86_64-linux-<br>
gnu/libKF5ItemModels.so /usr/lib/x86_64-linux-gnu/<a href="http://libpython2.7.so" \
rel="noreferrer" target="_blank">libpython2.7.so</a><br> <br>
That created the cxx/kitemmodels.so library<br>
<br>
$ ls cxx/<br>
kitemmodels.exp sipAPIkitemmodels.h<br>
kitemmodelsmod.sip.tmp sipkitemmodelscmodule.cpp<br>
kitemmodels.so* sipkitemmodelscmodule.o<br>
module.Makefile sipkitemmodelsKSelectionProxyModel.cpp<br>
module.sbf sipkitemmodelsKSelectionProxyModel.o<br>
pythontest.py*<br>
<br>
$ cat cxx/pythontest.py<br>
#!/usr/bin/env python<br>
<br>
from PyQt5 import QtCore<br>
<br>
import kitemmodels<br>
<br>
$ ./cxx/pythontest.py<br>
<span>Traceback (most recent call last):<br>
</span> File "./cxx/pythontest.py", line 5, in <module><br>
import kitemmodels<br>
SystemError: dynamic module not initialized properly<br>
<br>
<br>
Is there some other step to take before that import will \
work?<br></blockquote><div><br></div><div>Only to pick up the code I pushed earlier \
tonight [1]. This brings a slight change the in the CLI for the sip_compiler.py. The \
second argument ("cxx" above) is now optional and defaulted from the \
project rules "project_name". For KF5, the latter is PyKF5, and the net \
effect of the changes is that you get the following by default:<br><br></div><div>- a \
directory PyKF5<br></div><div>- containing a __init__.py<br></div><div>- and all the \
.so files such as KItemModels.so<br></div><div>- and a disposable tmp \
directory<br><br></div><div>so that you can then just say "from PyKF5 import \
KItemModels". You'll notice that in support of this, there is a temporary \
list of hardcoded libraries to link against (a quick hack just to get us \
going).<br></div><div><br></div><div>Thanks, Shaheed<br><br></div><div>[1] See <a \
href="https://www.riverbankcomputing.com/pipermail/pyqt/2016-April/037331.html">https://www.riverbankcomputing.com/pipermail/pyqt/2016-April/037331.html</a> \
for the details.<br><br></div><div> </div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"> <div><div>
Thanks,<br>
<br>
Steve.<br>
_______________________________________________<br>
Kde-bindings mailing list<br>
<a href="mailto:Kde-bindings@kde.org" target="_blank">Kde-bindings@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/kde-bindings" rel="noreferrer" \
target="_blank">https://mail.kde.org/mailman/listinfo/kde-bindings</a><br> \
</div></div></blockquote></div><br></div></div>
[Attachment #6 (text/plain)]
_______________________________________________
Kde-bindings mailing list
Kde-bindings@kde.org
https://mail.kde.org/mailman/listinfo/kde-bindings
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic