[prev in list] [next in list] [prev in thread] [next in thread]
List: pypy-dev
Subject: Re: [pypy-dev] cppyy use case question
From: Alex Pyattaev <alex.pyattaev () gmail ! com>
Date: 2012-05-31 8:33:34
Message-ID: 1543846.T9Sm08LvAA () hunter-laptop
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
keskiviikko 30 toukokuu 2012 18:52:25 wlavrijsen@lbl.gov kirjoitti:
> Hi Alex,
>
> > So, cpp_class acts as a container for python objects, and it does not need
> > to know what kind of object is inside. I was able to make it work with
> > pypy through basic C API, using SWIG and typemaps by mapping void* into
> > PyObject*, but that is somewhat hackish. I am now thinking how would the
> > same logic look with cppyy extension.
>
> this should be supported directly (PyROOT does), but where in CPython
> handing a PyObject* is trivial, in PyPy one has to be created. My best
> guess is that the route would be cpyext to get such a PyObject*, then hand
> that over.
>
> I don't think it can be hacked around (a void* argument will peel the
> contained C++ object, which will fail, since it's a real Python object).
>
> Identity preservation (on the return) should be easy, and the C++ side
> should keep the refcount up for the proper duration.
>
> Let me figure that one out. Is as good a reason as any to have some fun and
> do some coding again for the first time in weeks. :P
>
Thanks! I have accumulated some understanding about the process, and the issue
can be worked around by having a dictionary that would map ints into python
objects and vice versa before those are sent to c++, so when c++ functions
feth them back they can be retreived. However, this would incur certain
overhead, which may be rather annoying. Anyway, the whole point was to test if
it will be any faster then cpyext, so unless this feature is properly resolved
the test does not make any massive amount of sense (mostly due to this extra
overhead). I'll probably run it anyway though=)
Thanks for looking into it, its good to hear that somebody cares!
> > PS:
> > building cppyy is pain in the ass... You guys seriously need to think
> > about
> > build logs =)
>
> Sorry (but this is why at CERN we have a global file system (afs) and simply
> install there for all users :) ) ... So, is there anything in the
> instructions that was missing and needs to be added?
>
With respect to build, I installed root from gentoo repository, and the
problem there is that headers are in /usr/include/root and the actual libs are
in /usr/lib/root, but the pypy build expects libs to be in
/usr/include/root/lib if you specify SYSROOT=/usr/include/root. So instead I
ended up having to symlink those together to replicate whatever pypy build was
expecting. This is indeed a hax, but probably if you install root from source
package it is not needed. Anyway, it should be benefitial to add a header and
library check in the beginning of the build to be certain that it does not
fail after 20 mins of compiling. But for now symlink works just fine.
With respect to genreflex command, you could probably warn people that they
should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible
with newer versions of STL, and crashes on standard headers. Also, any headers
that are not absolutely necessary to define the interface to Python should be
removed if possible, that does make life a lot easier, even with SWIG, because
it makes wrappers smaller.
Best Regards,
Alex
PS: If you want I could give you benchmark results of cppyy vs cpyext+SWIG
when we get it running, probably it will be of some interest. I could give you
the entire code, but it would not be much use since it requires a ton of other
libraries.
[Attachment #5 (unknown)]
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" \
"http://www.w3.org/TR/REC-html40/strict.dtd"> <html><head><meta name="qrichtext" \
content="1" /><style type="text/css"> p, li { white-space: pre-wrap; }
</style></head><body style=" font-family:'Terminus'; font-size:11pt; font-weight:400; \
font-style:normal;"> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">keskiviikko \
30 toukokuu 2012 18:52:25 wlavrijsen@lbl.gov kirjoitti:</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> Hi Alex,</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > So, cpp_class acts \
as a container for python objects, and it does not need</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > to know what kind \
of object is inside. I was able to make it work with</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> > pypy through basic C API, using SWIG \
and typemaps by mapping void* into</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> > PyObject*, but that is somewhat hackish. I am now \
thinking how would the</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> > same logic look with cppyy extension.</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> this should be supported \
directly (PyROOT does), but where in CPython</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> handing a PyObject* is trivial, in PyPy one \
has to be created. My best</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> guess is that the route would be cpyext to get such a \
PyObject*, then hand</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> that over.</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> </p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> I don't think it can be hacked around (a void* argument will \
peel the</p> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> \
contained C++ object, which will fail, since it's a real Python object).</p> <p \
style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> Identity preservation \
(on the return) should be easy, and the C++ side</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> should keep the refcount up for the proper \
duration.</p> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p> \
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> Let me figure that one \
out. Is as good a reason as any to have some fun and</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> do some coding again for the first time in \
weeks. :P</p> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p> \
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Thanks! I have accumulated \
some understanding about the process, and the issue can be worked around by having a \
dictionary that would map ints into python objects and vice versa before those are \
sent to c++, so when c++ functions feth them back they can be retreived. However, \
this would incur certain overhead, which may be rather annoying. Anyway, the whole \
point was to test if it will be any faster then cpyext, so unless this feature is \
properly resolved the test does not make any massive amount of sense (mostly due to \
this extra overhead). I'll probably run it anyway though=)</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Thanks for looking into it, \
its good to hear that somebody cares!</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> > PS:</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> > building cppyy is pain in the ass... \
You guys seriously need to think</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> > about</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> > build logs =)</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> </p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> Sorry (but this is why at CERN we have a \
global file system (afs) and simply</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">> install there for all users :) ) ... So, is there anything in \
the</p> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> \
instructions that was missing and needs to be added?</p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">> </p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">With respect to build, I installed root from \
gentoo repository, and the problem there is that headers are in /usr/include/root and \
the actual libs are in /usr/lib/root, but the pypy build expects libs to be in \
/usr/include/root/lib if you specify SYSROOT=/usr/include/root. So instead I ended up \
having to symlink those together to replicate whatever pypy build was expecting. This \
is indeed a hax, but probably if you install root from source package it is not \
needed. Anyway, it should be benefitial to add a header and library check in the \
beginning of the build to be certain that it does not fail after 20 mins of \
compiling. But for now symlink works just fine.</p> <p \
style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">With respect to genreflex \
command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz \
gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes \
on standard headers. Also, any headers that are not absolutely necessary to define \
the interface to Python should be removed if possible, that does make life a lot \
easier, even with SWIG, because it makes wrappers smaller.</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Best Regards,</p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Alex</p> <p \
style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">PS: If you want I could give \
you benchmark results of cppyy vs cpyext+SWIG when we get it running, probably it \
will be of some interest. I could give you the entire code, but it would not be much \
use since it requires a ton of other libraries.</p></body></html>
_______________________________________________
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic