[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openembedded-core
Subject:    [OE-core] [PATCH 2/2] sstate/staging: Handle directory creation race issue
From:       "Richard Purdie" <richard.purdie () linuxfoundation ! org>
Date:       2021-06-30 12:12:14
Message-ID: 20210630121214.1965373-2-richard.purdie () linuxfoundation ! org
[Download RAW message or body]

Content-Transfer-Encoding: 8bit

The sstate code tries to be careful about racing around directory creation.
In particular, the copyhardlinktree code creates the directory tree first
allowing for "already exists" errors and ignoring them, then hardlinks the
files in.

Unfortunately the sstate removal code can race against this since it
will try and remove empty directories. If there is some bad timing,
a newly created directory can be removed before it was populated, leading
to build failures.

We could try and add locking but this would damage performance, we've been
there before. It is also unclear where to actually place locks just based on
the contents of a manifest file which may cover multiple sstate install
locations for a given task.

Instead, lets disable directory removal in the problematic "shared" core
path. This could result in a few more empty directories being left on disk
but those should be harmless and better than locking hurting performance
or rare build races.

[YOCTO #13999]
[YOCTO #14379]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass  | 8 +++++---
 meta/classes/staging.bbclass | 6 +++---
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 72eebd70c91..6c8e64ed00d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -483,7 +483,7 @@ def sstate_clean_cachefiles(d):
         ss = sstate_state_fromvars(ld, task)
         sstate_clean_cachefile(ss, ld)
 
-def sstate_clean_manifest(manifest, d, prefix=None):
+def sstate_clean_manifest(manifest, d, canrace=False, prefix=None):
     import oe.path
 
     mfile = open(manifest)
@@ -501,7 +501,9 @@ def sstate_clean_manifest(manifest, d, prefix=None):
             if entry.endswith("/"):
                 if os.path.islink(entry[:-1]):
                     os.remove(entry[:-1])
-                elif os.path.exists(entry) and len(os.listdir(entry)) == 0:
+                elif os.path.exists(entry) and len(os.listdir(entry)) == 0 and not \
canrace: +                    # Removing directories whilst builds are in progress \
exposes a race. Only +                    # do it in contexts where it is safe to do \
so.  os.rmdir(entry[:-1])
             else:
                 os.remove(entry)
@@ -539,7 +541,7 @@ def sstate_clean(ss, d):
         for lock in ss['lockfiles']:
             locks.append(bb.utils.lockfile(lock))
 
-        sstate_clean_manifest(manifest, d)
+        sstate_clean_manifest(manifest, d, canrace=True)
 
         for lock in locks:
             bb.utils.unlockfile(lock)
diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 806a85773af..29005ac0b62 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -409,7 +409,7 @@ python extend_recipe_sysroot() {
         if os.path.islink(f) and not os.path.exists(f):
             bb.note("%s no longer exists, removing from sysroot" % f)
             lnk = os.readlink(f.replace(".complete", ""))
-            sstate_clean_manifest(depdir + "/" + lnk, d, workdir)
+            sstate_clean_manifest(depdir + "/" + lnk, d, True, workdir)
             os.unlink(f)
             os.unlink(f.replace(".complete", ""))
 
@@ -454,7 +454,7 @@ python extend_recipe_sysroot() {
             fl = depdir + "/" + l
             bb.note("Task %s no longer depends on %s, removing from sysroot" % \
(mytaskname, l))  lnk = os.readlink(fl)
-            sstate_clean_manifest(depdir + "/" + lnk, d, workdir)
+            sstate_clean_manifest(depdir + "/" + lnk, d, True, workdir)
             os.unlink(fl)
             os.unlink(fl + ".complete")
 
@@ -475,7 +475,7 @@ python extend_recipe_sysroot() {
                 continue
             else:
                 bb.note("%s exists in sysroot, but is stale (%s vs. %s), removing." \
                % (c, lnk, c + "." + taskhash))
-                sstate_clean_manifest(depdir + "/" + lnk, d, workdir)
+                sstate_clean_manifest(depdir + "/" + lnk, d, True, workdir)
                 os.unlink(depdir + "/" + c)
                 if os.path.lexists(depdir + "/" + c + ".complete"):
                     os.unlink(depdir + "/" + c + ".complete")
-- 
2.30.2



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153443): https://lists.openembedded.org/g/openembedded-core/message/153443
Mute This Topic: https://lists.openembedded.org/mt/83890602/4454766
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [openembedded-core@marc.info]
-=-=-=-=-=-=-=-=-=-=-=-



[prev in list] [next in list] [prev in thread] [next in thread]