[prev in list] [next in list] [prev in thread] [next in thread] 

List:       quanta
Subject:    Re: [Quanta] Quanta and CPU/Memory consumption
From:       Eric Laffoon <eric () kdewebdev ! org>
Date:       2008-05-21 4:48:34
Message-ID: 200805202148.35104.eric () kdewebdev ! org
[Download RAW message or body]

On Tuesday 20 May 2008 5:50:34 pm James Ellis wrote:
> ok, here's some more info:
>
> Symptoms:
> * Quanta uses between 50 and 99% of CPU and about 4-7% of memory on
> startup, loading up medium to large projects (> 1000 files). The system is
> fine at other times.

As I mentioned, quite unusual... As was also mentioned 1000+ file projects are 
also not exactly common. I thought mine were bad with maybe 1000 and startup 
on my machine is about a second or two, taking the most time to load files 
from the last session.

> * Running an strace on the pid shows Quanta is doing various filesystem
> operations like fstat64, lstat64, chdir, stat64, open, readlink etc etc.
> For large projects with symlinks this can take 5 to 10 minutes to complete.
> Would it be correct to assume this is to populate the file tree, among
> other things ?

I would have to check with Andras as he is much more familiar with the code, 
or dig in myself. I don't have a lot of time for that, plus we're working 
onKDE4 versions. What I'd like to do is make sure not to port any problems 
first of all.
>
> Projects:
> * There are some projects I work on which I don't have control over the
> amount of files
> * Some projects are a split about 80/20 images/text
> * Some projects have symlinks to other filesystem locations
>
> System:
> * Intel(R) Pentium(R) M processor 1600MHz
> * 1.2GB RAM
> * Quanta 3.5.9
> * KDE 3.5.9 / 4.0.4 (happens in both)
> * Kubuntu 8.04, fully updated
> Other software running at the same time (that use similar resources)
> * Kontact, Xorg, Firefox.
>
> Ways I have tried to fix:
> * reorganise any symlinks to avoid possible recursion
> * alter the Project Properties to ignore symlinks and non text files (never
> open jpg's in Quanta, anyway) but these options don't seem to be available

You can set file filters based on extentions.
> * make sure no other processes are causing the problem, they aren't.
>
> My questions:
> * Is it possible to ignore symlinks per project ? Some symlinks are
> placeholders to other directories in the same project - to aid with web
> server setup. Symlinks could then be just denoted in the tree view rather
> than followed ?
> * Is it possible to have an option like "include only text files" or
> "exclude binary files" in the Project Properties, rather than the "Insert
> only markup, script and image files" and "Insert files with the following
> mask" in the New Project widget. Neither of the current options are valid
> for me, especially as files may be named with various extensions.
> * Could there be a minimal startup option - to just show files, directories
> and symlinks in the tree. I don't really mind if the icons are not correct
> for the file type or if the parser finds an error (I handle parser errors
> in various other tests). As long as I can get a file tree representation
> and click on the files to open them quickly, it's good enough for me.
> I noticed there are some other treeview plugins for Kate floating around
> the web - maybe one of those could be an option to investigate ?
>
> Any other info needed, feel free to ask me.
>
> Thanks
> james

Another question would be your actual management of the project. For instance, 
are you using the upload management in Quanta or are you using an RCS or 
other system? The point being, if you are using SVN to manage your project 
then technically you could also use just the file tree with topfolders and 
skip the use of the project.

In KDE4 one of the things I want to implement is the ability to set profiles 
that limit the exposure of files and interface components by user and task. 
These would be managed, so it would extend the abilities. In your case I 
suspect (and had from the start) it may be more an issue with the size of the 
project. It would be much faster for Andras to look at the code, and it's 
also possible things like the distribution internal selection of file 
management and configuration could be an issue. There are several programs 
that could be configured in a way to be redundant or create issues which 
would require a recompile, but I can't be at all sure if that's the case. 
Also the tree widget can begin to absorb resources and cycles with large data 
sets... However while this was in issue with Kommander and interfacing loops 
with QStrings it seems very unlikely it would be with Quanta.

If Andras can give some feedback or test with large projects that would help. 
I'll also ask independent of this thread on the list.
>
> On Mon, 19 May 2008 05:13:15 pm Eric Laffoon wrote:
> > On Sunday 18 May 2008 8:32:39 pm James Ellis wrote:
> > > Hi
> > >
> > > I'm not sure about anyone else but I find Quanta's CPU and memory usage
> > > to be pretty extreme, especially on startup. Out of all the KDE apps
> > > that I use it's the one that slows down the system the most, by a fair
> > > margin. To the point where there is a noticeable lag when typing this
> > > message between key stroked and the key appearing on the page.
> >
> > And then there is the old standard about diagnostic information... A man
> > goes to his doctor and says "I feel blue" the doctor says "take these
> > pills". Of course not! Now on your system I see you're running... Hmmm?
> > I've got GIGO here.
> >
> > > It's most noticeable on startup when a project is being scanned - it's
> > > less of an issue with small projects which would point to a scaling
> > > issue. I regularly see CPU usage between 50 and 99% and memory usage
> > > generally outstrips Xorg which usually runs at 6% or so. Once Quanta
> > > has finished it's startup dance, system load returns to normal.
> > > Doing an strace on the process reveals what looks to be Quanta doing a
> > > stat on every file in the project.
> >
> > One would think the people using Quanta would be crazy to put up with
> > developers nuts enough to write a program that bad, and I'm sure I'm not
> > alone in not seeing the same thing you are. I don't doubt you're seeing
> > it but you are describing an anomoly under the apparent assumption I'm
> > going to be seeing the same thing. You may as well ask about a movie I
> > didn't see... Diagnostics requres data.
> >
> > > So, I guess the big questions are
> > > - is this cpu and memory usage really needed?
> >
> > Absolutely not! However that's not to say various factors could not cause
> > isolated cases like you're describing and certainly without the ability
> > to evalute the factors of that data I just have to hope it's something
> > like you're running an old version where known bugs have been fixed or
> > really huge files with very little RAM.
> >
> > > - Does Quanta need to stat every file in the project on startup?
> >
> > Quanta parses every file on load. That's a little different and more
> > involved. It is essential to provide the services it offers. In all but
> > the most extreme cases it should not cause resource issues. I'm not sure
> > about the rest without looking at the code.
> >
> > > - Couldn't it just read the XML project file and put a representation
> > > of that in the file tree? This would put the onus on a developer who
> > > makes major file changes to rescan the tree manually.
> >
> > It does read the XML file. You're probably noting safety procedures to
> > insure the file is valid. Without knowing specifics you probably have
> > some other issue causing system load. The code you're referencing has
> > been stable for years with who knows how many thousands or millions of
> > users and it's been several years since I've seen any performance
> > complaints like this. Those were related to parser bugs with small
> > specific issues. Without knowing specifics I am sure that an operation
> > like checking a file is very fast and you would have to have an
> > uncharacteristicly huge number of files for that to be an issue.
> >
> > > - Maybe there is an option to turn it off for startup ?
> >
> > I would have to check with Andras. However unless your projects are
> > massive or your system ancient it is likely there is another problem and
> > without knowing the specifics there is too much to speculate on to offer
> > a good answer.
> >
> > > Thanks
> > > James
> >
> > Occasionally people do bring up specific issues where unusual
> > circumstances cause problems. Often other system issues cause problems,
> > but we rely on our users to get us a reliable description and sometimes
> > example files to recreate an anomoly and test.
>
> _______________________________________________
> Quanta mailing list
> Quanta@mail.kde.org
> https://mail.kde.org/mailman/listinfo/quanta



-- 
Eric Laffoon
Project Lead - kdewebdev module
_______________________________________________
Quanta mailing list
Quanta@mail.kde.org
https://mail.kde.org/mailman/listinfo/quanta
[prev in list] [next in list] [prev in thread] [next in thread]