'Re: [OE-core] [PATCH] devtool: modify: correctly handle multiple sources in SRC_URI'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openembedded-core
Subject:    Re: [OE-core] [PATCH] devtool: modify: correctly handle multiple sources in SRC_URI
From:       "Julien Stephan" <jstephan () baylibre ! com>
Date:       2023-09-30 9:59:02
Message-ID: CAEHHSvZsyVenNwXP_hyVa83kSVPr3xTOj2x2cSpztrx8EoG-KA () mail ! gmail ! com
[Download RAW message or body]

Le mer. 27 sept. 2023 Ă  22:36, Richard Purdie
<richard.purdie@linuxfoundation.org> a ĂŠcrit :
> 
> On Wed, 2023-09-27 at 18:50 +0200, Julien Stephan wrote:
> > [YOCTO #15162]
> > 
> > When recipes such as bzip2 have several sources in SRC_URI, all the
> > sources are correctly extracted into devtool temporary workdir, but
> > currently, we only get back the primary source and all files declared
> > using "files://", then the devtool temporary workdir is deleted
> > 
> > Fix this by moving all remaining files from devtool temporary workdir
> > into actual WORKDIR (except "source-date-epoch", "recipe-sysroot",
> > and "recipe-sysroot-native") before deleting it
> > 
> > Here is the stack trace from devtool modify/build bzip2:
> > 
> > NOTE: bzip2: compiling from external source tree \
> >                 <...>/build/workspace/sources/bzip2
> > ERROR: bzip2-1.0.8-r0 do_install_ptest_base: \
> > ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', \
> >                 1, None, None)
> > ERROR: Logfile of failure stored in: \
> > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
> >  Log data follows:
> > > DEBUG: Executing shell function do_install_ptest_base
> > > NOTE: make -j 16 \
> > > DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest \
> > > install-ptest sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
> > > ../../../../../../workspace/sources/bzip2/Makefile.am      > \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
> > >  cp ../../../../../../workspace/sources/bzip2/sample1.ref      \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ \
> > > cp ../../../../../../workspace/sources/bzip2/sample2.ref      \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ \
> > > cp ../../../../../../workspace/sources/bzip2/sample3.ref      \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ \
> > > cp ../../../../../../workspace/sources/bzip2/sample1.bz2      \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ \
> > > cp ../../../../../../workspace/sources/bzip2/sample2.bz2      \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ \
> > > cp ../../../../../../workspace/sources/bzip2/sample3.bz2      \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ \
> > > ln -s /usr/bin/bzip2          \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
> > >                 
> > > cp: cannot stat \
> > > '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No \
> > >                 such file or directory
> > > WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 \
> > > exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/
> > 1.0.8/git/commons-compress \
> > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
> > 
> > > WARNING: Backtrace (BB generated script):
> > > #1: do_install_ptest, \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, \
> > > line 189 #2: do_install_ptest_base, \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, \
> > > line 158 #3: main, \
> > > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, \
> > > line 226
> > ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) \
> >                 failed with exit code '1'
> > NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and \
> > 1 failed. 
> > Summary: 1 task failed:
> > <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base
> > 
> > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > ---
> > scripts/lib/devtool/standard.py | 8 ++++++++
> > 1 file changed, 8 insertions(+)
> > 
> > diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> > index d53fb810071..6c8ad7a6c8c 100644
> > --- a/scripts/lib/devtool/standard.py
> > +++ b/scripts/lib/devtool/standard.py
> > @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, \
> > config, basepath, works shutil.move(srcsubdir, srctree)
> > symlink_oelocal_files_srctree(d,srctree)
> > 
> > +            # In case SRC_URI contains multiple sources
> > +            # they are extracted into the devtool temporary workdir
> > +            # We need to move them into WORKDIR, otherwise they will be lost
> > +            for path in os.listdir(os.path.join(tempdir, 'workdir')):
> > +                if not path in ["source-date-epoch", "recipe-sysroot", \
> > "recipe-sysroot-native"]: +                    workdir = d.getVar('WORKDIR')
> > +                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
> > +
> > if is_kernel_yocto:
> > logger.info('Copying kernel config to srctree')
> > shutil.copy2(os.path.join(tempdir, '.config'), srctree)
> 
> Looks like a good start although I worry a bit about the hardcoded list
> of exceptions.


Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can
easily get the list of secondary sources?

Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if
the destination file/directory already exists
* retrieve SRC_URI and manually build the list of secondary sources to
move. This looks a little bit overkill here...
What do you think?

> 
> Is it possible to add a test case to the devtool testsuite so this
> can't regress in future?

Sure, I'll do it!

Cheers
Juliem
> 


sure I will add a test for that.


> Thanks,
> 
> Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188455): https://lists.openembedded.org/g/openembedded-core/message/188455
Mute This Topic: https://lists.openembedded.org/mt/101621289/4454766
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [openembedded-core@marc.info]
-=-=-=-=-=-=-=-=-=-=-=-



[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic