[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kdevelop-devel
Subject:    Re: On Memory consumption of KDevelop-PG
From:       David Nolden <zwabel () googlemail ! com>
Date:       2009-12-19 17:45:03
Message-ID: 200912191845.03718.zwabel () googlemail ! com
[Download RAW message or body]

Am Samstag 19 Dezember 2009 17:36:59 schrieb Alexander Dymo:
> субота, 19-гру-2009 11:49:47 Andreas Pakulat ви написали:
> > IMHO its no big deal if the parser needs some 200M for parsing a single
> > file as long as afterwards the memory is free again for re-use. In
> > particular it must not be fragmented (which should already work due to
> > usage of the memory pool). For how to make the AST smaller see further
> > down on your "compress the AST" question.
> 
> Well, but now huge parser memory consumption is an issue (at least for
>  C++). I have 3.5M C file which is parsed into 550M AST.
> 550M isn't that small amount of memory...
> 
> And that's just one file. Try parsing Ruby language sources - you'll end up
> using 3G of memory for parsing. That's definitely wrong. Not everybody has
> this much of memory.
> 
> There're actually bugs in c++ parser where it creates AST nodes which
>  aren't used (added to the tree). I have a patch (not public yet, still
>  breaks some duchain tests) for that which reduces the memory consumption
>  in my test case from 550M to 400M. But that's all I can do without doing
>  AST compression or removing members from AST classes.

There is a few things we can do in future that would have a huge impact:
- Replace size_t with uint everywhere within the parser
- Move the "AST* parent" and "DUContext* ducontext" members out of the AST, 
into a separate map in ParseSession, as they are not needed in most cases.
- Do the same thing with the "comments" member of CommentAST

Together, these would probably reduce the size of the AST by 2/3.

However I consider this all that important atm, because the performance needs 
to be considered carefully when doing some of these changes, and KDevelop4 
right now anyway needs a fairly modern machine, which makes the memory-
requirements not all that fatal. The more important issues right now are 
stability and usability.

And btw. how did you measure those 550MB? It sounds very much..

-- 
KDevelop-devel mailing list
KDevelop-devel@kdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinfo/kdevelop-devel

[prev in list] [next in list] [prev in thread] [next in thread]