From cmake Thu Mar 21 04:10:36 2019 From: Kuba Ober Date: Thu, 21 Mar 2019 04:10:36 +0000 To: cmake Subject: Re: [CMake] CMake Project Generation Speedup Message-Id: <51184E18-F25D-4A9C-A67F-A8CE7A73E1E6 () mareimbrium ! org> X-MARC-Message: https://marc.info/?l=cmake&m=155314144521241 MIME-Version: 1 Content-Type: multipart/mixed; boundary="--===============1142513404==" --===============1142513404== Content-Type: multipart/alternative; boundary=Apple-Mail-40F70F7A-F192-488E-9F60-CF24FC5833B6 Content-Transfer-Encoding: 7bit --Apple-Mail-40F70F7A-F192-488E-9F60-CF24FC5833B6 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable The best advice I had for myself: profile the code and tweak it. The last I l= ooked, and that was a while ago, the generators could use a threaded write q= ueue so that the file output would overlap with file generation. Those proba= bly could be memory mapped files, too, to further minimize the overhead: wri= ting to mapped pages that are dumped to disk via DMA (that=E2=80=99s how pag= e flushes work) is cheaper than copying all the data in little chunks into a= small buffer in the file output code, and then once again into the page tha= t ends up flushed to disk. Never mind that the write latencies are on the ho= t path, where they don=E2=80=99t belong. If generators could be const-correct (not writing anywhere into the Makefile= data structures), they could all be parallelized as well =E2=80=93 but I=E2= =80=99m not sure whether that=E2=80=99s a minor hack or a major rework.=20 Cheers, Kuba > 20 mars 2019 kl. 16:53 skrev J. Caleb Wherry : >=20 > Did anything ever come of this? >=20 > I am in a similar boat: we have >800 targets on our full build (native C++= , Managed C++, C#, Java, CUDA, etc) and the majority of the time for the con= figure/generate steps takes place in the generate step (>70%). >=20 > I understand there is a lot of IO since the generate step has to write the= project files and filters for each C++ project (the majority of our project= s) for VS generators (what we use). I'm just looking to see if there is anyt= hing to look at or potentially speedup up the generate step besides "get a f= aster drive". >=20 > Thanks! > -Caleb >=20 >> On Thu, Nov 17, 2016 at 11:15 AM Damian wrote= : >> Hi all, >>=20 >> We are still in the process of switching our large Make-based build to CM= ake. One of the issues we're running into is the time it takes to reparse an= d regenerate the CMake project (whether ninja, VS, or make) after touching a= ny CMake file. To give you an idea, we have about 1000 targets and that take= s a good 2 min for CMake to rerun. >>=20 >> Are there any plans to speed this up? Maybe parallelize it in some way or= do a better job regenerating only what needs regenerating? Is there anythin= g we can do on our side to reduce our regeneration times? >>=20 >> For example, if using a VS generator, each directory in the source that h= as a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch o= ne of those CMakeLists.txt, only that .sln/.vcproj would get regenerated. >>=20 >> Thanks for any help. >> --=20 >>=20 >> Powered by www.kitware.com >>=20 >> Please keep messages on-topic and check the CMake FAQ at: http://www.cmak= e.org/Wiki/CMake_FAQ >>=20 >> Kitware offers various services to support the CMake community. For more i= nformation on each offering, please visit: >>=20 >> CMake Support: http://cmake.org/cmake/help/support.html >> CMake Consulting: http://cmake.org/cmake/help/consulting.html >> CMake Training Courses: http://cmake.org/cmake/help/training.html >>=20 >> Visit other Kitware open-source projects at http://www.kitware.com/openso= urce/opensource.html >>=20 >> Follow this link to subscribe/unsubscribe: >> http://public.kitware.com/mailman/listinfo/cmake >=20 >=20 > --=20 > J. Caleb Wherry > Scientific Software Engineer >=20 > http://www.calebwherry.com > +1 (615) 708-5651 > calebwherry@gmail.com > --=20 >=20 > Powered by www.kitware.com >=20 > Please keep messages on-topic and check the CMake FAQ at: http://www.cmake= .org/Wiki/CMake_FAQ >=20 > Kitware offers various services to support the CMake community. For more i= nformation on each offering, please visit: >=20 > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html >=20 > Visit other Kitware open-source projects at http://www.kitware.com/opensou= rce/opensource.html >=20 > Follow this link to subscribe/unsubscribe: > https://cmake.org/mailman/listinfo/cmake --Apple-Mail-40F70F7A-F192-488E-9F60-CF24FC5833B6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
The= best advice I had for myself: profile the code and tweak it. The last I loo= ked, and that was a while ago, the generators could use a threaded write que= ue so that the file output would overlap with file generation. Those probabl= y could be memory mapped files, too, to further minimize the overhead: writi= ng to mapped pages that are dumped to disk via DMA (that=E2=80=99s how page f= lushes work) is cheaper than copying all the data in little chunks into a sm= all buffer in the file output code, and then once again into the page that e= nds up flushed to disk. Never mind that the write latencies are on the hot p= ath, where they don=E2=80=99t belong.

If generators could be const-correct (not writing anywhere into t= he Makefile data structures), they could all be parallelized as well =E2=80=93= but I=E2=80=99m not sure whether that=E2=80=99s a minor hack or a major rew= ork. 

Cheers, Kuba

20 mars 2019 kl. 16:53 skrev J. Caleb Wherry <calebwherry@gmail.com>:

<= /div>
Did anythin= g ever come of this?

I am in a similar boat: we have >= 800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc)= and the majority of the time for the configure/generate steps takes place i= n the generate step (>70%).

I understand there i= s a lot of IO since the generate step has to write the project files and fil= ters for each C++ project (the majority of our projects) for VS generators (= what we use). I'm just looking to see if there is anything to look at or pot= entially speedup up the generate step besides "get a faster drive".

Thanks!
-Caleb

On Thu, Nov 17, 2016 at 11:15 A= M Damian <damian.campeanu@gm= ail.com> wrote:
Hi all,

We are still in th= e process of=20 switching our large Make-based build to CMake. One of the issues we're=20 running into is the time it takes to reparse and regenerate the CMake=20 project (whether ninja, VS, or make) after touching any CMake file. To=20 give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up?=20 Maybe parallelize it in some way or do a better job regenerating only=20 what needs regenerating? Is there anything we can do on our side to=20 reduce our regeneration times?

For example, if using a VS=20 generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those=20 CMakeLists.txt, only that .sln/.vcproj would get regenerated.

T= hanks for any help.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://ww= w.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more inf= ormation on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.ht= ml
CMake Training Courses: http://cmake.org/cmake/help/training.= html

Visit other Kitware open-source projects at http://ww= w.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake


--
J. Caleb Wherry
Scientific Software Eng= ineer

<= /a>
http://www.= calebwherry.com
+1 (615) 708-5651
calebwherry@gmail.c= om
--

Powered by w= ww.kitware.com

Please keep messages on-= topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more i= nformation on each offering, please visit:

= CMake Support: http://c= make.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/he= lp/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/trainin= g.html

Visit other Kitware open-source p= rojects at htt= p://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinf= o/cmake
= --Apple-Mail-40F70F7A-F192-488E-9F60-CF24FC5833B6-- --===============1142513404== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake --===============1142513404==--