[prev in list] [next in list] [prev in thread] [next in thread]
List: cmake
Subject: Re: [CMake] CMake Project Generation Speedup
From: Kuba Ober <kuba () mareimbrium ! org>
Date: 2019-03-21 4:10:36
Message-ID: 51184E18-F25D-4A9C-A67F-A8CE7A73E1E6 () mareimbrium ! org
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
The best advice I had for myself: profile the code and tweak it. The last I looked, \
and that was a while ago, the generators could use a threaded write queue so that the \
file output would overlap with file generation. Those probably could be memory mapped \
files, too, to further minimize the overhead: writing to mapped pages that are dumped \
to disk via DMA (that's how page flushes work) is cheaper than copying all the data \
in little chunks into a small buffer in the file output code, and then once again \
into the page that ends up flushed to disk. Never mind that the write latencies are \
on the hot path, where they don't belong.
If generators could be const-correct (not writing anywhere into the Makefile data \
structures), they could all be parallelized as well – but I'm not sure whether \
that's a minor hack or a major rework.
Cheers, Kuba
> 20 mars 2019 kl. 16:53 skrev J. Caleb Wherry <calebwherry@gmail.com>:
>
> Did anything ever come of this?
>
> I am in a similar boat: we have >800 targets on our full build (native C++, Managed \
> C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate \
> steps takes place in the generate step (>70%).
> I understand there is a lot of IO since the generate step has to write the project \
> files and filters for each C++ project (the majority of our projects) for VS \
> generators (what we use). I'm just looking to see if there is anything to look at \
> or potentially speedup up the generate step besides "get a faster drive".
> Thanks!
> -Caleb
>
> > On Thu, Nov 17, 2016 at 11:15 AM Damian <damian.campeanu@gmail.com> wrote:
> > Hi all,
> >
> > We are still in the process of switching our large Make-based build to CMake. One \
> > of the issues we're running into is the time it takes to reparse and regenerate \
> > the CMake project (whether ninja, VS, or make) after touching any CMake file. To \
> > give you an idea, we have about 1000 targets and that takes a good 2 min for \
> > CMake to rerun.
> > Are there any plans to speed this up? Maybe parallelize it in some way or do a \
> > better job regenerating only what needs regenerating? Is there anything we can do \
> > on our side to reduce our regeneration times?
> > For example, if using a VS generator, each directory in the source that has a \
> > CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of \
> > those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
> > Thanks for any help.
> > --
> >
> > Powered by www.kitware.com
> >
> > Please keep messages on-topic and check the CMake FAQ at: \
> > http://www.cmake.org/Wiki/CMake_FAQ
> > Kitware offers various services to support the CMake community. For more \
> > information on each offering, please visit:
> > CMake Support: http://cmake.org/cmake/help/support.html
> > CMake Consulting: http://cmake.org/cmake/help/consulting.html
> > CMake Training Courses: http://cmake.org/cmake/help/training.html
> >
> > Visit other Kitware open-source projects at \
> > http://www.kitware.com/opensource/opensource.html
> > Follow this link to subscribe/unsubscribe:
> > http://public.kitware.com/mailman/listinfo/cmake
>
>
> --
> J. Caleb Wherry
> Scientific Software Engineer
>
> http://www.calebwherry.com
> +1 (615) 708-5651
> calebwherry@gmail.com
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: \
> http://www.cmake.org/Wiki/CMake_FAQ
> Kitware offers various services to support the CMake community. For more \
> information on each offering, please visit:
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at \
> http://www.kitware.com/opensource/opensource.html
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
[Attachment #5 (text/html)]
<html><head><meta http-equiv="content-type" content="text/html; \
charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">The best \
advice I had for myself: profile the code and tweak it. The last I looked, and that \
was a while ago, the generators could use a threaded write queue so that the file \
output would overlap with file generation. Those probably could be memory mapped \
files, too, to further minimize the overhead: writing to mapped pages that are dumped \
to disk via DMA (that's how page flushes work) is cheaper than copying all the data \
in little chunks into a small buffer in the file output code, and then once again \
into the page that ends up flushed to disk. Never mind that the write latencies are \
on the hot path, where they don't belong.</div><div dir="ltr"><br></div><div \
dir="ltr">If generators could be const-correct (not writing anywhere into the \
Makefile data structures), they could all be parallelized as well – but I'm not \
sure whether that's a minor hack or a major rework. </div><div \
dir="ltr"><br></div><div dir="ltr">Cheers, Kuba</div><div dir="ltr"><br>20 mars 2019 \
kl. 16:53 skrev J. Caleb Wherry <<a \
href="mailto:calebwherry@gmail.com">calebwherry@gmail.com</a>>:<br><br></div><blockquote \
type="cite"><div dir="ltr"><div dir="ltr">Did anything ever come of \
this?<div><br></div><div>I am in a similar boat: we have >800 targets on our full \
build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for \
the configure/generate steps takes place in the generate step \
(>70%).</div><div><br></div><div>I understand there is a lot of IO since the \
generate step has to write the project files and filters for each C++ project (the \
majority of our projects) for VS generators (what we use). I'm just looking to see if \
there is anything to look at or potentially speedup up the generate step besides "get \
a faster drive".</div><div><br></div><div>Thanks!</div><div>-Caleb</div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 17, 2016 at 11:15 \
AM Damian <<a href="mailto:damian.campeanu@gmail.com">damian.campeanu@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div><div><div>Hi all,<br><br></div>We are still in the process of \
switching our large Make-based build to CMake. One of the issues we're running into \
is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, \
or make) after touching any CMake file. To give you an idea, we have about 1000 \
targets and that takes a good 2 min for CMake to rerun.<br><br></div>Are there any \
plans to speed this up? Maybe parallelize it in some way or do a better job \
regenerating only what needs regenerating? Is there anything we can do on our side \
to reduce our regeneration times?<br><br>For example, if using a VS
generator, each directory in the source that has a CMakeLists.txt gets a
.vcproj and .sln generated. Ideally, if I touch one of those
CMakeLists.txt, only that .sln/.vcproj would get regenerated.<br><br></div>Thanks for \
any help.</div>
-- <br>
<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" \
target="_blank">www.kitware.com</a><br> <br>
Please keep messages on-topic and check the CMake FAQ at: <a \
href="http://www.cmake.org/Wiki/CMake_FAQ" rel="noreferrer" \
target="_blank">http://www.cmake.org/Wiki/CMake_FAQ</a><br> <br>
Kitware offers various services to support the CMake community. For more information \
on each offering, please visit:<br> <br>
CMake Support: <a href="http://cmake.org/cmake/help/support.html" rel="noreferrer" \
target="_blank">http://cmake.org/cmake/help/support.html</a><br> CMake Consulting: <a \
href="http://cmake.org/cmake/help/consulting.html" rel="noreferrer" \
target="_blank">http://cmake.org/cmake/help/consulting.html</a><br> CMake Training \
Courses: <a href="http://cmake.org/cmake/help/training.html" rel="noreferrer" \
target="_blank">http://cmake.org/cmake/help/training.html</a><br> <br>
Visit other Kitware open-source projects at <a \
href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" \
target="_blank">http://www.kitware.com/opensource/opensource.html</a><br> <br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/cmake" rel="noreferrer" \
target="_blank">http://public.kitware.com/mailman/listinfo/cmake</a></blockquote></div><br \
clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div>J. \
Caleb Wherry</div><i>Scientific Software Engineer</i><div><br><a \
href="http://www.calebwherry.com" target="_blank"></a><div><a \
href="http://www.calebwherry.com" \
target="_blank">http://www.calebwherry.com</a><br><span><span title="Call with Google \
Voice"><span title="Call with Google Voice">+1 (615) \
708-5651</span></span></span><div><a href="mailto:calebwherry@gmail.com" \
target="_blank">calebwherry@gmail.com</a></div></div></div></div> \
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>-- \
</span><br><span></span><br><span>Powered by <a \
href="http://www.kitware.com">www.kitware.com</a></span><br><span></span><br><span>Please \
keep messages on-topic and check the CMake FAQ at: <a \
href="http://www.cmake.org/Wiki/CMake_FAQ">http://www.cmake.org/Wiki/CMake_FAQ</a></span><br><span></span><br><span>Kitware \
offers various services to support the CMake community. For more information on each \
offering, please visit:</span><br><span></span><br><span>CMake Support: <a \
href="http://cmake.org/cmake/help/support.html">http://cmake.org/cmake/help/support.html</a></span><br><span>CMake \
Consulting: <a href="http://cmake.org/cmake/help/consulting.html">http://cmake.org/cmake/help/consulting.html</a></span><br><span>CMake \
Training Courses: <a \
href="http://cmake.org/cmake/help/training.html">http://cmake.org/cmake/help/training.html</a></span><br><span></span><br><span>Visit \
other Kitware open-source projects at <a \
href="http://www.kitware.com/opensource/opensource.html">http://www.kitware.com/opensource/opensource.html</a></span><br><span></span><br><span>Follow \
this link to subscribe/unsubscribe:</span><br><span><a \
href="https://cmake.org/mailman/listinfo/cmake">https://cmake.org/mailman/listinfo/cmake</a></span><br></div></blockquote></body></html>
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at: \
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more information \
on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at \
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic