[prev in list] [next in list] [prev in thread] [next in thread] 

List:       doxygen-users
Subject:    Re: [Doxygen-users] HAVE_DOT performance issue
From:       Dimitri van Heesch <doxygen () gmail ! com>
Date:       2013-07-30 18:11:43
Message-ID: 6FF4427C-295F-44B1-B0DB-E267D64F86C7 () gmail ! com
[Download RAW message or body]


On Jul 30, 2013, at 19:49 , Tom Udale <tom@ionoptix.com> wrote:

> Greetings,
> 
> One of my favorite things about doxygen is all the cool images which show
> various relationships between classes when you enable HAVE_DOT. 
> Unfortunately on my rather large project, turning on HAVE_DOT makes doxygen
> take _forever_ to build.  My belief is that it is because of the number of
> files that are sitting in the root of the html directory afterward.  This
> happens in spite of having enabled CREATE_SUBDIRS.
> 
> Here are the relevant statistics for two builds where the only change is the
> state of HAVE_DOT.  Both are into an empty destination directory with
> doxygen 1.8.4:
> 
> HAVE_DOT=FALSE:  
> 
> Time to build: 00:09:16.46
> 
> 
> HTML directory contents: 1632 items (including directories) 921 javasript;
> 65 png files; 626 xhtml files;
> 
> 
> HAVE_DOT=TRUE: 
> 
> Time to build: 02:35:18.45
> 
> Intermediate HTML directory contents (while building): 13049 items
> (including directories).  5249 DOT files;  902 javascript; 5249 MD5 files.
> 
> Final HTML directory contents: 18,297 items (including directories) 921
> javasript; 5248 map files; 5249 md5 files; 5314 png files; 1545 xhtml files;

These are the totals including the contents of sub-directories, or only the root
directory?

> 
> 
> Yes, that is 2 hours and 26 minutes longer when HAVE_DOT=TRUE.
> 
> If I do the math for the added amount of time and my 5249 dot files, I get
> something like 1.67 secs/dot file.
> 
> This is tremendously slow, particularly considering that I am running on a
> 12 core system (and indeed there are 12 instances of dot running); the
> single core equivalent is 20 seconds per file.
> 
> It almost has to be the fact that the file system is wading through these
> 13k-18k items while running dot (I guess possibly up to 24k when both the
> dot and png files are in there together toward the end).
> 
> As a test, I copied the first 1000 inherit_graph.dot files into their own
> directory and ran dot on them via a batch file.

batch file? so you have a 12 core system running Windows?
I think I found the real bottleneck ;-)
Have you ever tried the same experiment using Linux with 
the ext4 or ext3 filesystem? 

> 
> Here are the results for that:
> 
> first 1000 dots alone = 40.14 sec = .0401 sec/file
> first 91 dots alone =3.24 secs = .035 sec/file
> 
> This is all with a single instance of dot, so it is comparable to the 20
> second value above.
> 
> If I were to run my batch file on the 5249 dot files, I would expect it to
> take 209 seconds (3.5 minutes) with a single core (assuming 0.04s per file)
> or 17 seconds with 12 instances (assuming no interactions between
> instances).
> 
> This is a far cry from the 2 hours and 20 minutes.
> 
> I would imagine what is needed here is to pull more of the output into the
> CREATE_DIRS structure and possibly to put the dot files themselves into some
> temporary subdirectory (tree?).  
> 
> This is easy to say of course, but perhaps not too difficult to do also?

Can you send me a listing of the top level directory, so I can see which files
are there? that might help me improve things...

Regards,
  Dimitri



------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent 
caught up. So what steps can you take to put your SQL databases under 
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
_______________________________________________
Doxygen-users mailing list
Doxygen-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/doxygen-users
[prev in list] [next in list] [prev in thread] [next in thread]