[prev in list] [next in list] [prev in thread] [next in thread] 

List:       opensuse-factory
Subject:    Re: some late build chain changes in factory degrades build performance of some packages
From:       Hans-Peter Jansen <hpj () urpla ! net>
Date:       2022-11-12 17:37:07
Message-ID: 14775414.tv2OnDr8pf () xrated
[Download RAW message or body]

Am Samstag, 12. November 2022, 16:56:35 CET schrieb Jan Engelhardt:
> On Friday 2022-11-11 20:56, Hans-Peter Jansen wrote:
> >some late changes in the build chain degrades build performance to a crawl:
> >e.g. a package typically builds in under two minutes:
> >
> >https://build.opensuse.org/package/live_build_log/vdr:plugins/vdr-plugin-li
> >ve/ openSUSE_Tumbleweed/x86_64
> >
> >shows this now:
> >
> >[ 2624s] CC live.o
> >[ 3161s] CC thread.o
> 
> 1. `osc build ...` in one terminal
> 
> 2. Second terminal,
> 
> 	strace -fe execve -p 24582
> 
> 	(pid of make)
> 
> and I observed that make re-runs pkg-config a damn lot.
> 
> What recently changed in openSUSE is the make program (4.3 -> 4.4).
> 
> >From make's changelog, I identify two changes that seem like they
> 
> could be relevant:
> 
> 	https://lists.gnu.org/archive/html/info-gnu/2022-10/msg00008.html
> 
> 	>Previously each target in a explicit grouped target rule was
> 	>considered individually: if the targets needed by the build
> 	>were not out of date the recipe was not run even if other
> 	>targets in the group were out of date
> 
> 	or
> 
> 	>Previously makefile variables marked as export were not
> 	>exported to commands started by the $(shell ...) function.
> 	>Now, all exported variables are exported to $(shell ...).
> 
> Of course, these alone do not constitute problem - we have over 1000
> packages that exercise `make` one way or the other, but they do so
> through a proper means of Makefile generators like configure or cmake
> or something of the sort.
> 
> 3. So we're looking at a shitty Makefile.
> 
> 	vdr$ make -ddd clean
> 	Reading makefile 'Makefile'...
> 	grep: warning: stray \ before #
> 	Makefile:18: not recursively expanding PKGCFG to export to shell 
function
> 	Makefile:18: not recursively expanding PKGCFG to export to shell 
function
> 	Makefile:18: not recursively expanding PKGCFG to export to shell 
function
> 	Makefile:18: not recursively expanding PKGCFG to export to shell 
function
> 	Reading makefile 'global.mk' (search path) (no ~ expansion)...
> 	Makefile:27: not recursively expanding CFLAGS to export to shell 
function
> 	Makefile:18: not recursively expanding PKGCFG to export to shell 
function
> 	Makefile:18: not recursively expanding PKGCFG to export to shell 
function
> 	...
> 
> Just look at the mess:
> 
> 	PKGCFG = $(if $(VDRDIR),$(shell pkg-config --variable=$(1)
> $(VDRDIR)/vdr.pc),$(shell PKG_CONFIG_PATH="$$PKG_CONFIG_PATH:../../.."
> pkg-config --variable=$(1) vd... LIBDIR = $(call PKGCFG,libdir)
> 	LOCDIR = $(call PKGCFG,locdir)
> 	PLGCFG = $(call PKGCFG,plgcfg)
> 	RESDIR = $(call PKGCFG,resdir)
> 
> 	### The compiler options:
> 	export CFLAGS   = $(call PKGCFG,cflags)
> 	export CXXFLAGS = $(call PKGCFG,cxxflags)
> 
> 	...
> 
> 	ifeq ($(TNTNET-CONFIG),)
> 	TNTVERSION = $(shell pkg-config --modversion tntnet | sed -e's/\.//
g' | sed
> -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"}
> /^....$ CXXFLAGS  += $(shell pkg-config --cflags tntnet)
> 	LIBS      += $(shell pkg-config --libs tntnet)
> 	else
> 	TNTVERSION = $(shell tntnet-config --version | sed -e's/\.//g' | 
sed
> -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"}
> /^....$$/ { pr CXXFLAGS  += $(shell tntnet-config --cxxflags)
> 	LIBS      += $(shell tntnet-config --libs)
> 
> So remember what the changelog said: export all variables whenever
> $(shell ..) is run. The irony is that, to set up the environment for
> that shell run, it has to expand PKGCFG, which involves calling
> at least another $(shell ...).
> 
> And that multiplies fast.
> 
> *yuck*

First at all, thanks Jan! Your analysis is much appreciated..

Oh my, wth.

This reveals a bunch of questions:
 * How to address this mess? 
   Probably preparing the environment before in order to avoid those shell
   expansions within the Makefile will cut it
 * The new make behaviour is kind of questionable, 
   * since it can be easily tricked into a DoS, since it might behave like a
     fork bomb now
   * feels like a regression nevertheless

Reminds me of a saying:

You don't understand recursion unless you understand recursion!

Thanks again,
Pete
[prev in list] [next in list] [prev in thread] [next in thread]