[prev in list] [next in list] [prev in thread] [next in thread]
List: pypy-dev
Subject: Re: [pypy-dev] Great experience with PyPy
From: Gelin Yan <dynamicgl () gmail ! com>
Date: 2013-02-07 16:00:58
Message-ID: CABkOF6TVTLE-Y-fF2E3uFeWj7A+__MK4WF0D238V=RKanEMYZw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Thu, Feb 7, 2013 at 11:28 PM, Костя Лопухин
<konstantin.lopuhin@chtd.ru>wrote:
> PyPy supports postgres with either psycopg2cffi or psycopg2-ctypes
> bindings. We use psycopg2cffi in production (and maintain them), and
> here http://chtd.ru/blog/bystraya-rabota-s-postgres-pod-pypy/?lang=en
> are some benchmarks.
> And yes, PyPy is cool :) Typically giving 3x speedups, and some memory
> savings sometimes.
>
> 2013/2/7 Gelin Yan <dynamicgl@gmail.com>:
> >
> >
> > On Thu, Feb 7, 2013 at 10:11 PM, Phyo Arkar <phyo.arkarlwin@gmail.com>
> > wrote:
> >>
> >> Pypy should have a page for "Success Stories!"
> >>
> >> Now with this and Quora proving Power of PyPy , i am beginning to start
> >> converting my projects into PyPy soon!
> >> I am only withholding right now because my projects uses a lot of C
> >> Libraries and Numpy/Matplotlib/scilit-learn.
> >>
> >> Thanks
> >>
> >> Phyo.
> >>
> >> On Thursday, February 7, 2013, Maciej Fijalkowski wrote:
> >>>
> >>> On Thu, Feb 7, 2013 at 1:55 PM, Marko Tasic <mtasic85@gmail.com>
> wrote:
> >>> > Hi,
> >>> >
> >>> > I would like to share short story with you and share what we have
> >>> > accomplished with PyPy and its friends so far.
> >>> >
> >>> > Company that I have worked for last 7 months (intentionally unnamed)
> >>> > gave me absolute permission to pick up technologies on which we based
> >>> > our solution. What we do is: crawl for PDFs and newspapers articles,
> >>> > download, translate them if needed, OCR if needed, do extensive
> >>> > analysis of downloaded PDFs and articles, store them in more
> organized
> >>> > structures for faster querying, search for them and generate bunch of
> >>> > complex reports.
> >>> >
> >>> > From very beginning I decided to go with PyPy no matter what. What we
> >>> > picked is following:
> >>> > * Flask for web framework, and few of its extensions such as
> >>> > Flask-Login, Flask-Principal, Flask-WTF, Flask-Mail, etc.
> >>> > * Cassandra as database because of its features and great experience
> >>> > with it. PyCassa is used as client to talk to Cassandra server.
> >>> > * ElasticSearch as distributed search engine, and its client library
> >>> > pyes.
> >>> > * Whoosh as search engine, but with some modifications to support
> >>> > Cassandra as storage and distributed locking.
> >>> > * Redis, and its client library redis-py, for caching and to speed up
> >>> > common auto-completion patterns.
> >>> > * ZooKeeper, and its client library Kazoo, for distributed locking
> >>> > which plays essential role in system for transaction-like behavior
> >>> > over many services at once.
> >>> > * Celery in conjunction with RabbitMQ for task distribution.
> >>> > * Sentry for error logging.
> >>> >
> >>> > What we have developed on our own are wrappers and clients for:
> >>> > * Moses which is language translator
> >>> > * Tesseract which is OCR engine
> >>> > * Cassandra store for Whoosh
> >>> > * wkhtmltopdf and wkhtmltoimage which are used for conversion of HTML
> >>> > to PDF/Image
> >>> > * etc
> >>> >
> >>> > Now when product is finished and in final testing phase, I can say
> >>> > that we did not regret because we used PyPy and stack around it.
> >>> > Typical speed improvement is 2x-3x over CPython in our case, but
> >>> > anyway we are mostly IO and memory bound, expect for Celery workers
> >>> > where we do analysis which are again many small CPU intensive tasks
> >>> > that are exchanged via RabbitMQ. Another reason why we don't see
> >>> > speedup us is that we are dependent on external software (servers)
> >>> > written in Erlang and Java.
> >>> >
> >>> > I'm already planing to do Cassandra (distributed key/value only
> >>> > database without index features), ZooKeeper, Redis and ElasticSearch
> >>> > ports in Python for next projects, and hopefully opensource them.
> >>> >
> >>> > Regards,
> >>> > Marko Tasic
> >>> > _______________________________________________
> >>> > pypy-dev mailing list
> >>> > pypy-dev@python.org
> >>> > http://mail.python.org/mailman/listinfo/pypy-dev
> >>>
> >>> Awesome!
> >>>
> >>> I'm glad people can make pypy work for non-trivial tasks which require
> >>> a lot of dependencies. We're trying to lower the bar, however it takes
> >>> time.
> >>>
> >>> Cheers,
> >>> fijal
> >>> _______________________________________________
> >>> pypy-dev mailing list
> >>> pypy-dev@python.org
> >>> http://mail.python.org/mailman/listinfo/pypy-dev
> >>
> >>
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev@python.org
> >> http://mail.python.org/mailman/listinfo/pypy-dev
> >>
> >
> >
> > Hi, It might be off topic. I want to know whether pypy support postgres.
> The
> > last time I noticed ctypes based psycopg2 was still beta. I mainly use
> > twisted & postgres. pypy supports twisted well but not good for psycopg2.
> >
> > Regards
> >
> > gelin yan
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev@python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> >
>
>
>
> --
> Константин Лопухин, разра отчик
> Компания ЧТД -- http://chtd.ru
> +7 (495) 646-87-45, до авочный 333
>
Hi
Glad to hear that. I will give it a try. By the way, Can i use it on
windows? It looks like cffi support windows.
Regards
gelin yan
[Attachment #5 (text/html)]
<br><br><div class="gmail_quote">On Thu, Feb 7, 2013 at 11:28 PM, Костя Лопухин <span \
dir="ltr"><<a href="mailto:konstantin.lopuhin@chtd.ru" \
target="_blank">konstantin.lopuhin@chtd.ru</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> PyPy supports postgres with either psycopg2cffi or \
psycopg2-ctypes<br> bindings. We use psycopg2cffi in production (and maintain them), \
and<br> here <a href="http://chtd.ru/blog/bystraya-rabota-s-postgres-pod-pypy/?lang=en" \
target="_blank">http://chtd.ru/blog/bystraya-rabota-s-postgres-pod-pypy/?lang=en</a><br>
are some benchmarks.<br>
And yes, PyPy is cool :) Typically giving 3x speedups, and some memory<br>
savings sometimes.<br>
<br>
2013/2/7 Gelin Yan <<a \
href="mailto:dynamicgl@gmail.com">dynamicgl@gmail.com</a>>:<br> <div><div \
class="h5">><br> ><br>
> On Thu, Feb 7, 2013 at 10:11 PM, Phyo Arkar <<a \
href="mailto:phyo.arkarlwin@gmail.com">phyo.arkarlwin@gmail.com</a>><br> > \
wrote:<br> >><br>
>> Pypy should have a page for "Success Stories!"<br>
>><br>
>> Now with this and Quora proving Power of PyPy , i am beginning to start<br>
>> converting my projects into PyPy soon!<br>
>> I am only withholding right now because my projects uses a lot of C<br>
>> Libraries and Numpy/Matplotlib/scilit-learn.<br>
>><br>
>> Thanks<br>
>><br>
>> Phyo.<br>
>><br>
>> On Thursday, February 7, 2013, Maciej Fijalkowski wrote:<br>
>>><br>
>>> On Thu, Feb 7, 2013 at 1:55 PM, Marko Tasic <<a \
href="mailto:mtasic85@gmail.com">mtasic85@gmail.com</a>> wrote:<br> >>> \
> Hi,<br> >>> ><br>
>>> > I would like to share short story with you and share what we \
have<br> >>> > accomplished with PyPy and its friends so far.<br>
>>> ><br>
>>> > Company that I have worked for last 7 months (intentionally \
unnamed)<br> >>> > gave me absolute permission to pick up technologies on \
which we based<br> >>> > our solution. What we do is: crawl for PDFs and \
newspapers articles,<br> >>> > download, translate them if needed, OCR if \
needed, do extensive<br> >>> > analysis of downloaded PDFs and articles, \
store them in more organized<br> >>> > structures for faster querying, \
search for them and generate bunch of<br> >>> > complex reports.<br>
>>> ><br>
>>> > From very beginning I decided to go with PyPy no matter what. What \
we<br> >>> > picked is following:<br>
>>> > * Flask for web framework, and few of its extensions such as<br>
>>> > Flask-Login, Flask-Principal, Flask-WTF, Flask-Mail, etc.<br>
>>> > * Cassandra as database because of its features and great \
experience<br> >>> > with it. PyCassa is used as client to talk to \
Cassandra server.<br> >>> > * ElasticSearch as distributed search engine, \
and its client library<br> >>> > pyes.<br>
>>> > * Whoosh as search engine, but with some modifications to \
support<br> >>> > Cassandra as storage and distributed locking.<br>
>>> > * Redis, and its client library redis-py, for caching and to speed \
up<br> >>> > common auto-completion patterns.<br>
>>> > * ZooKeeper, and its client library Kazoo, for distributed \
locking<br> >>> > which plays essential role in system for \
transaction-like behavior<br> >>> > over many services at once.<br>
>>> > * Celery in conjunction with RabbitMQ for task distribution.<br>
>>> > * Sentry for error logging.<br>
>>> ><br>
>>> > What we have developed on our own are wrappers and clients for:<br>
>>> > * Moses which is language translator<br>
>>> > * Tesseract which is OCR engine<br>
>>> > * Cassandra store for Whoosh<br>
>>> > * wkhtmltopdf and wkhtmltoimage which are used for conversion of \
HTML<br> >>> > to PDF/Image<br>
>>> > * etc<br>
>>> ><br>
>>> > Now when product is finished and in final testing phase, I can \
say<br> >>> > that we did not regret because we used PyPy and stack \
around it.<br> >>> > Typical speed improvement is 2x-3x over CPython in \
our case, but<br> >>> > anyway we are mostly IO and memory bound, expect \
for Celery workers<br> >>> > where we do analysis which are again many \
small CPU intensive tasks<br> >>> > that are exchanged via RabbitMQ. \
Another reason why we don't see<br> >>> > speedup us is that we are \
dependent on external software (servers)<br> >>> > written in Erlang and \
Java.<br> >>> ><br>
>>> > I'm already planing to do Cassandra (distributed key/value \
only<br> >>> > database without index features), ZooKeeper, Redis and \
ElasticSearch<br> >>> > ports in Python for next projects, and hopefully \
opensource them.<br> >>> ><br>
>>> > Regards,<br>
>>> > Marko Tasic<br>
>>> > _______________________________________________<br>
>>> > pypy-dev mailing list<br>
>>> > <a href="mailto:pypy-dev@python.org">pypy-dev@python.org</a><br>
>>> > <a href="http://mail.python.org/mailman/listinfo/pypy-dev" \
target="_blank">http://mail.python.org/mailman/listinfo/pypy-dev</a><br> \
>>><br> >>> Awesome!<br>
>>><br>
>>> I'm glad people can make pypy work for non-trivial tasks which \
require<br> >>> a lot of dependencies. We're trying to lower the bar, \
however it takes<br> >>> time.<br>
>>><br>
>>> Cheers,<br>
>>> fijal<br>
>>> _______________________________________________<br>
>>> pypy-dev mailing list<br>
>>> <a href="mailto:pypy-dev@python.org">pypy-dev@python.org</a><br>
>>> <a href="http://mail.python.org/mailman/listinfo/pypy-dev" \
target="_blank">http://mail.python.org/mailman/listinfo/pypy-dev</a><br> >><br>
>><br>
>> _______________________________________________<br>
>> pypy-dev mailing list<br>
>> <a href="mailto:pypy-dev@python.org">pypy-dev@python.org</a><br>
>> <a href="http://mail.python.org/mailman/listinfo/pypy-dev" \
target="_blank">http://mail.python.org/mailman/listinfo/pypy-dev</a><br> >><br>
><br>
><br>
> Hi, It might be off topic. I want to know whether pypy support postgres. The<br>
> last time I noticed ctypes based psycopg2 was still beta. I mainly use<br>
> twisted & postgres. pypy supports twisted well but not good for \
psycopg2.<br> ><br>
> Regards<br>
><br>
> gelin yan<br>
><br>
> _______________________________________________<br>
> pypy-dev mailing list<br>
> <a href="mailto:pypy-dev@python.org">pypy-dev@python.org</a><br>
> <a href="http://mail.python.org/mailman/listinfo/pypy-dev" \
target="_blank">http://mail.python.org/mailman/listinfo/pypy-dev</a><br> ><br>
<br>
<br>
<br>
</div></div>--<br>
Константин Лопухин, разра отчик<br>
Компания ЧТД -- <a href="http://chtd.ru" target="_blank">http://chtd.ru</a><br>
<a href="tel:%2B7%20%28495%29%20646-87-45" value="+74956468745">+7 (495) \
646-87-45</a>, до авочный 333<br> \
</blockquote></div><br><div>Hi</div><div> </div><div> Glad to hear that. I will \
give it a try. By the way, Can i use it on windows? It looks like cffi support \
windows.</div><div><br></div><div>Regards</div><div><br></div> <div>gelin yan</div>
_______________________________________________
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic