[prev in list] [next in list] [prev in thread] [next in thread]
List: pypy-dev
Subject: Re: [pypy-dev] Great experience with PyPy
From: Marko Tasic <mtasic85 () gmail ! com>
Date: 2013-02-08 11:22:25
Message-ID: CAE3+++Tp_Mn6yp6ZHKXQ-wq1ueF66BNRxNgeOa_mUTjKYKS5bQ () mail ! gmail ! com
[Download RAW message or body]
Thanks everyone for support.
@fijal
In one of previous emails, I already told you that I'll be using pypy
for real-life problems on medium to large scale projects. It is also
very hard convincing companies that they should invest money in open
source, but as far as they contribute to open source projects, I'm
also satisfied in some way. Anyway, I prefer money ;)
@carl
I'm very bad at writing blog posts, but would like to explain in email
what we have done, what obstacles have we faced and how we solved
them.
@armin
Because I don't care of speed (I already have plenty of CPU cores not
used all the time), and I only care of correctness and maintainability
of code, your STM will perfectly fit in our requirements. As far as i
know, every developer working on serious large scale project after
going over your STM descriptions (emails and blogs) gives me the same
answer about it, and that is the perfect solution for "per machine"
concurrent programming. As far as I have freedom to pick technologies,
I will definitely relay on it on one of the next projects. What is the
status of it ATM, and what is best way to test and deploy pypy with
stm?
Regards,
Marko Tasic
On Thu, Feb 7, 2013 at 5:08 PM, Костя Лопухин <kostia.lopuhin@gmail.com> wrote:
> Hi! I did not test it on Windows, there may be problems with
> installation (searching for postgres header files, the config is not
> very smart - https://github.com/chtd/psycopg2cffi/blob/master/psycopg2cffi/_impl/libpq.py#L209),
> but they should be solvable I hope - submit a bug if you have
> problems.
>
> 2013/2/7 Gelin Yan <dynamicgl@gmail.com>:
>>
>>
>> On Thu, Feb 7, 2013 at 11:28 PM, Костя Лопухин <konstantin.lopuhin@chtd.ru>
>> wrote:
>>>
>>> PyPy supports postgres with either psycopg2cffi or psycopg2-ctypes
>>> bindings. We use psycopg2cffi in production (and maintain them), and
>>> here http://chtd.ru/blog/bystraya-rabota-s-postgres-pod-pypy/?lang=en
>>> are some benchmarks.
>>> And yes, PyPy is cool :) Typically giving 3x speedups, and some memory
>>> savings sometimes.
>>>
>>> 2013/2/7 Gelin Yan <dynamicgl@gmail.com>:
>>> >
>>> >
>>> > On Thu, Feb 7, 2013 at 10:11 PM, Phyo Arkar <phyo.arkarlwin@gmail.com>
>>> > wrote:
>>> >>
>>> >> Pypy should have a page for "Success Stories!"
>>> >>
>>> >> Now with this and Quora proving Power of PyPy , i am beginning to start
>>> >> converting my projects into PyPy soon!
>>> >> I am only withholding right now because my projects uses a lot of C
>>> >> Libraries and Numpy/Matplotlib/scilit-learn.
>>> >>
>>> >> Thanks
>>> >>
>>> >> Phyo.
>>> >>
>>> >> On Thursday, February 7, 2013, Maciej Fijalkowski wrote:
>>> >>>
>>> >>> On Thu, Feb 7, 2013 at 1:55 PM, Marko Tasic <mtasic85@gmail.com>
>>> >>> wrote:
>>> >>> > Hi,
>>> >>> >
>>> >>> > I would like to share short story with you and share what we have
>>> >>> > accomplished with PyPy and its friends so far.
>>> >>> >
>>> >>> > Company that I have worked for last 7 months (intentionally unnamed)
>>> >>> > gave me absolute permission to pick up technologies on which we
>>> >>> > based
>>> >>> > our solution. What we do is: crawl for PDFs and newspapers articles,
>>> >>> > download, translate them if needed, OCR if needed, do extensive
>>> >>> > analysis of downloaded PDFs and articles, store them in more
>>> >>> > organized
>>> >>> > structures for faster querying, search for them and generate bunch
>>> >>> > of
>>> >>> > complex reports.
>>> >>> >
>>> >>> > From very beginning I decided to go with PyPy no matter what. What
>>> >>> > we
>>> >>> > picked is following:
>>> >>> > * Flask for web framework, and few of its extensions such as
>>> >>> > Flask-Login, Flask-Principal, Flask-WTF, Flask-Mail, etc.
>>> >>> > * Cassandra as database because of its features and great experience
>>> >>> > with it. PyCassa is used as client to talk to Cassandra server.
>>> >>> > * ElasticSearch as distributed search engine, and its client library
>>> >>> > pyes.
>>> >>> > * Whoosh as search engine, but with some modifications to support
>>> >>> > Cassandra as storage and distributed locking.
>>> >>> > * Redis, and its client library redis-py, for caching and to speed
>>> >>> > up
>>> >>> > common auto-completion patterns.
>>> >>> > * ZooKeeper, and its client library Kazoo, for distributed locking
>>> >>> > which plays essential role in system for transaction-like behavior
>>> >>> > over many services at once.
>>> >>> > * Celery in conjunction with RabbitMQ for task distribution.
>>> >>> > * Sentry for error logging.
>>> >>> >
>>> >>> > What we have developed on our own are wrappers and clients for:
>>> >>> > * Moses which is language translator
>>> >>> > * Tesseract which is OCR engine
>>> >>> > * Cassandra store for Whoosh
>>> >>> > * wkhtmltopdf and wkhtmltoimage which are used for conversion of
>>> >>> > HTML
>>> >>> > to PDF/Image
>>> >>> > * etc
>>> >>> >
>>> >>> > Now when product is finished and in final testing phase, I can say
>>> >>> > that we did not regret because we used PyPy and stack around it.
>>> >>> > Typical speed improvement is 2x-3x over CPython in our case, but
>>> >>> > anyway we are mostly IO and memory bound, expect for Celery workers
>>> >>> > where we do analysis which are again many small CPU intensive tasks
>>> >>> > that are exchanged via RabbitMQ. Another reason why we don't see
>>> >>> > speedup us is that we are dependent on external software (servers)
>>> >>> > written in Erlang and Java.
>>> >>> >
>>> >>> > I'm already planing to do Cassandra (distributed key/value only
>>> >>> > database without index features), ZooKeeper, Redis and ElasticSearch
>>> >>> > ports in Python for next projects, and hopefully opensource them.
>>> >>> >
>>> >>> > Regards,
>>> >>> > Marko Tasic
>>> >>> > _______________________________________________
>>> >>> > pypy-dev mailing list
>>> >>> > pypy-dev@python.org
>>> >>> > http://mail.python.org/mailman/listinfo/pypy-dev
>>> >>>
>>> >>> Awesome!
>>> >>>
>>> >>> I'm glad people can make pypy work for non-trivial tasks which require
>>> >>> a lot of dependencies. We're trying to lower the bar, however it takes
>>> >>> time.
>>> >>>
>>> >>> Cheers,
>>> >>> fijal
>>> >>> _______________________________________________
>>> >>> pypy-dev mailing list
>>> >>> pypy-dev@python.org
>>> >>> http://mail.python.org/mailman/listinfo/pypy-dev
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> pypy-dev mailing list
>>> >> pypy-dev@python.org
>>> >> http://mail.python.org/mailman/listinfo/pypy-dev
>>> >>
>>> >
>>> >
>>> > Hi, It might be off topic. I want to know whether pypy support postgres.
>>> > The
>>> > last time I noticed ctypes based psycopg2 was still beta. I mainly use
>>> > twisted & postgres. pypy supports twisted well but not good for
>>> > psycopg2.
>>> >
>>> > Regards
>>> >
>>> > gelin yan
>>> >
>>> > _______________________________________________
>>> > pypy-dev mailing list
>>> > pypy-dev@python.org
>>> > http://mail.python.org/mailman/listinfo/pypy-dev
>>> >
>>>
>>>
>>>
>>> --
>>> Константин Лопухин, разработчик
>>> Компания ЧТД -- http://chtd.ru
>>> +7 (495) 646-87-45, добавочный 333
>>
>>
>> Hi
>>
>> Glad to hear that. I will give it a try. By the way, Can i use it on
>> windows? It looks like cffi support windows.
>>
>> Regards
>>
>> gelin yan
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev@python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic