[prev in list] [next in list] [prev in thread] [next in thread]
List: sbcl-devel
Subject: Re: [Sbcl-devel] run-program (spawn) + threading deadlock
From: Stas Boukarev <stassats () gmail ! com>
Date: 2018-12-15 16:32:02
Message-ID: CAF63=13Zgn2CM_cRa+=ZtXXpQYyOc1gTXxNaEzjor3LZokLeXA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Here's the test case:
https://bugs.launchpad.net/sbcl/+bug/1808641
On Wed, Nov 28, 2018 at 6:46 AM Stas Boukarev <stassats@gmail.com> wrote:
> There's no calls to run-program here, I need a test case that it involves
> run-program, or there is no way to know what's fixed and what's not.
>
> On Wed, Nov 28, 2018 at 6:20 AM Spenser Truex <struex0@gmail.com> wrote:
>
> > (defun deadlock (&optional (join 0))
> > (let ((mutex nil))
> > (let ((c (sb-thread:make-thread
> > (lambda ()
> > (setf mutex t)
> > (sleep 1)
> > (setf mutex nil))))
> > (a (sb-thread:make-thread
> > (lambda ()
> > (sb-impl::without-gcing (loop until (not mutex))))))
> > (b (sb-thread:make-thread (lambda () (gc)))))
> > (sb-thread:join-thread (elt (list a b c) join)))))
> > Change on line 3:
> > (defun |"fixed"| (&optional (join 0))
> > (let ((mutex nil))
> > (let ((c (sb-thread:make-thread
> > (lambda ()
> > (sb-impl::without-gcing (setf mutex t)
> > (sleep 1)
> > (setf mutex nil)))))
> > (a (sb-thread:make-thread
> > (lambda ()
> > (sb-impl::without-gcing (loop until (not mutex))))))
> > (b (sb-thread:make-thread (lambda () (gc)))))
> > (sb-thread:join-thread (elt (list a b c) join)))))
> > Obviously not having to go into SBCL internals just to use run-program
> > would be better. I'm curious why the without-gcing form is wrapped around
> > it like that.
> >
> >
> > On Tue, Nov 27, 2018 at 2:48 PM Stas Boukarev <stassats@gmail.com> wrote:
> >
> > > I need a test case.
> > >
> > > On Wed, Nov 28, 2018 at 12:32 AM Spenser Truex <struex0@gmail.com>
> > > wrote:
> > >
> > > >
> > > > The docstring for without-gcing in sysmacs.lisp line 35 is pretty clear
> > > > about the problem and what to do (assuming your description is accurate):
> > > > "Should be used with great care, and not at all in multithreaded
> > > > application
> > > > code: Any locks that are ever acquired while GC is inhibited need to be
> > > > always
> > > > held with GC inhibited to prevent deadlocks: if T1 holds the lock and is
> > > > stopped for GC while T2 is waiting for the lock inside WITHOUT-GCING the
> > > > system will be deadlocked. Since SBCL does not currently document its
> > > > internal
> > > > locks, application code can never be certain that this invariant is
> > > > maintained."
> > > > Not being able to use run-program reliably in multi-threaded code is a
> > > > bit of a bummer. Following the workaround described above, the way to fix
> > > > the deadlock is:
> > > > 1. use without-gcing for thread C before it locks malloc or any lock
> > > > that any other without-gcing thread (like A) would need.
> > > >
> > > > On Tue, Nov 27, 2018 at 8:48 AM Stas Boukarev <stassats@gmail.com>
> > > > wrote:
> > > >
> > > > > Is there a test-case?
> > > > >
> > > > > On Wed, Nov 21, 2018 at 3:40 AM JTK <jetmonk@gmail.com> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > >
> > > > > >
> > > > > > My understanding on this issue is imperfect, but I'd like to revisit
> > > > > > the old issue here to resolve some questions at the end of this post:
> > > > > >
> > > > > >
> > > > > > https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4D4B7966-8CFA-4457-B90C-6B272CABA644%40gmail.com/#msg34883875
> > > > > > <https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4D4B7966-8CFA-4457-B90C-6B272CABA644@gmail.com/#msg34883875>
> > > > > >
> > > > > > My SBCL version is 1.4.5.debian
> > > > > >
> > > > > > I have a multithread process that runs unix-subprocesses. Every so
> > > > > > often, it freezes. gdb shows one process to be
> > > > > > stuck in a malloc lock, after the fork() call.
> > > > > >
> > > > > > The other threads are stuck like this
> > > > > >
> > > > > > =========gdb output=========
> > > > > > #0 0x00007ffff79c26d6 in futex_abstimed_wait_cancelable (private=0,
> > > > > > abstime=0x0, expected=0, futex_word=0x7ffff41b8078) at
> > > > > > ../sysdeps/unix/sysv/linux/futex-internal.h:205
> > > > > > 205 in ../sysdeps/unix/sysv/linux/futex-internal.h
> > > > > > (gdb) backtrace
> > > > > > #0 0x00007ffff79c26d6 in futex_abstimed_wait_cancelable (private=0,
> > > > > > abstime=0x0, expected=0, futex_word=0x7ffff41b8078) at
> > > > > > ../sysdeps/unix/sysv/linux/futex-internal.h:205
> > > > > > #1 do_futex_wait (sem=sem@entry=0x7ffff41b8078, abstime=0x0) at
> > > > > > sem_waitcommon.c:111
> > > > > > #2 0x00007ffff79c27c8 in __new_sem_wait_slow (sem=0x7ffff41b8078,
> > > > > > abstime=0x0) at sem_waitcommon.c:181
> > > > > > #3 0x000000000041ee54 in os_sem_wait ()
> > > > > > #4 0x0000000000425373 in wait_for_thread_state_change ()
> > > > > > #5 0x000000000041c93d in sig_stop_for_gc_handler ()
> > > > > > #6 0x000000000041afc0 in ?? ()
> > > > > > #7 <signal handler called>
> > > > > > #8 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:37
> > > > > > #9 0x0000000000427df0 in futex_wait ()
> > > > > > #10 0x0000000021d488f4 in ?? ()
> > > > > > ============================
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Here's how I interpret it:
> > > > > >
> > > > > > **1. Thread A enters common-lisp run-program, where it does (see
> > > > > > code/run-program.lisp line 833)
> > > > > >
> > > > > > ======== code/run-program.lisp line 833 =====
> > > > > > (without-gcing
> > > > > > (spawn progname args-vec
> > > > > > stdin stdout stderr
> > > > > > (if search 1 0)
> > > > > > environment-vec pty-name
> > > > > > (if wait 1 0) directory))
> > > > > > =====================================
> > > > > >
> > > > > > **2. Thread B stars a garbage collection, sending the other threads a
> > > > > > GC interrupt.
> > > > > >
> > > > > > **3. Thread C happens to be holding a malloc lock, but it is
> > > > > > interrupted by thread B, so is sitting idle waiting for the GC to finish.
> > > > > >
> > > > > > **4. Thread A was sitting on the malloc lock too, but it isn't woken
> > > > > > interrupted by thread B because it is inside without-gcing,
> > > > > >
> > > > > > **4. Deadlock, as thread A waits forever for C to release the malloc
> > > > > > lock, while preventing B and C from doing the GC that would
> > > > > > allow C to return and release the malloc lock.
> > > > > >
> > > > > >
> > > > > >
> > > > > > It seems to me that without-gcing in run-program is the problem in
> > > > > > this deadlock, as would any foreign function call that does a malloc
> > > > > > inside without-gcing
> > > > > >
> > > > > > So my questions are:
> > > > > >
> > > > > >
> > > > > > 1. Is the without-gcing around spawn in (defun run-program …) in
> > > > > > code/run-program.lisp really necessary? I've taken it out in a patched
> > > > > > version of
> > > > > > SBCL and it seems to help with this problem (though it is not a
> > > > > > frequent problem, so I can't say if it is a cure).
> > > > > >
> > > > > > 2. Is it always a bad idea to mix spawn/fork with multiple threads in
> > > > > > SBCL? Python groups suggested that this was a bad idea in general, yet
> > > > > > it's a feature I need.
> > > > > >
> > > > > > 2a. If mixing spawn/fork with threads is a bad idea, then is there
> > > > > > a better way to do fork, like having a single-threaded server process
> > > > > > connected with a pipe
> > > > > > doing the forking and returning the results via a socket or
> > > > > > file descriptor?
> > > > > >
> > > > > > 3. Is it always a bad idea to put (without-gcing ..) around a foreign
> > > > > > function call, because this malloc lockup is not unique to spawn? (and \
> > > > > > there might be locks in C libraries besides malloc)
> > > > > >
> > > > > > 4. Also in (defun run-program …) in code/run-program.lisp , might it
> > > > > > be better to put (with-active-processes-lock ()
> > > > > > around only (push proc *active-processes*) and not around the big
> > > > > > spawn block, so that child processes don't delay each other
> > > > > > other across threads?
> > > > > > (see
> > > > > > https://github.com/sbcl/sbcl/blob/master/src/code/run-program.lisp,
> > > > > > line 821 at present date)
> > > > > >
> > > > > >
> > > > > > Many thanks for any tips, and apologies if I said anything totally
> > > > > > off the mark.
> > > > > >
> > > > > > J.Klein
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Sbcl-devel mailing list
> > > > > > Sbcl-devel@lists.sourceforge.net
> > > > > > https://lists.sourceforge.net/lists/listinfo/sbcl-devel
> > > > > >
> > > > > _______________________________________________
> > > > > Sbcl-devel mailing list
> > > > > Sbcl-devel@lists.sourceforge.net
> > > > > https://lists.sourceforge.net/lists/listinfo/sbcl-devel
> > > > >
> > > >
[Attachment #5 (text/html)]
<div dir="ltr">Here's the test case:<div><span style="font-size:13px"><a \
href="https://bugs.launchpad.net/sbcl/+bug/1808641">https://bugs.launchpad.net/sbcl/+bug/1808641</a></span><br></div></div><br><div \
class="gmail_quote"><div dir="ltr">On Wed, Nov 28, 2018 at 6:46 AM Stas Boukarev \
<<a href="mailto:stassats@gmail.com">stassats@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There's no calls \
to run-program here, I need a test case that it involves run-program, or there is no \
way to know what's fixed and what's not.</div><div dir="ltr"><br><div \
class="gmail_quote"><div dir="ltr">On Wed, Nov 28, 2018 at 6:20 AM Spenser Truex \
<<a href="mailto:struex0@gmail.com" target="_blank">struex0@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">(defun deadlock \
(&optional (join 0))<br> (let ((mutex nil))<br> (let \
((c (sb-thread:make-thread <br> (lambda () <br> \
(setf mutex t)<br> (sleep 1)<br> (setf \
mutex nil))))<br> (a (sb-thread:make-thread <br> \
(lambda () <br> (sb-impl::without-gcing (loop until (not \
mutex))))))<br> (b (sb-thread:make-thread (lambda () (gc)))))<br> \
(sb-thread:join-thread (elt (list a b c) join)))))<div>Change on line \
3:</div><div>(defun |"fixed"| (&optional (join 0))<br> (let \
((mutex nil))<br> (let ((c (sb-thread:make-thread <br> \
(lambda () <br> (sb-impl::without-gcing (setf mutex t)<br> \
(sleep 1)<br> (setf mutex nil)))))<br> \
(a (sb-thread:make-thread <br> (lambda () <br> \
(sb-impl::without-gcing (loop until (not mutex))))))<br> (b \
(sb-thread:make-thread (lambda () (gc)))))<br> \
(sb-thread:join-thread (elt (list a b c) join)))))<br></div><div>Obviously not having \
to go into SBCL internals just to use run-program would be better. I'm curious \
why the without-gcing form is wrapped around it like \
that.<br></div><div><br></div></div></div></div></div></div></div></div><br><div \
class="gmail_quote"><div dir="ltr">On Tue, Nov 27, 2018 at 2:48 PM Stas Boukarev \
<<a href="mailto:stassats@gmail.com" target="_blank">stassats@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I need a test \
case.<br><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 28, 2018 at 12:32 AM \
Spenser Truex <<a href="mailto:struex0@gmail.com" \
target="_blank">struex0@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div><br></div>The docstring \
for without-gcing in sysmacs.lisp line 35 is pretty clear about the problem and what \
to do (assuming your description is accurate):<br>"Should be used with great \
care, and not at all in multithreaded application<br>code: Any locks that are ever \
acquired while GC is inhibited need to be always<br>held with GC inhibited to prevent \
deadlocks: if T1 holds the lock and is<br>stopped for GC while T2 is waiting for the \
lock inside WITHOUT-GCING the<br>system will be deadlocked. Since SBCL does not \
currently document its internal<br>locks, application code can never be certain that \
this invariant is<br>maintained."</div><div>Not being able to use run-program \
reliably in multi-threaded code is a bit of a bummer. Following the workaround \
described above, the way to fix the deadlock is:</div><div>1. use without-gcing for \
thread C before it locks malloc or any lock that any other without-gcing thread (like \
A) would need.</div></div><div dir="ltr"><div><br></div><div dir="ltr"><div><div \
class="gmail_quote"><div dir="ltr">On Tue, Nov 27, 2018 at 8:48 AM Stas Boukarev \
<<a href="mailto:stassats@gmail.com" target="_blank">stassats@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Is \
there a test-case?</div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 21, \
2018 at 3:40 AM JTK <<a href="mailto:jetmonk@gmail.com" \
target="_blank">jetmonk@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div><div>Hello,</div><div><br></div><div><br></div><div><br></div><div>My \
understanding on this issue is imperfect, but I'd like to revisit the old issue here \
to resolve some questions at the end of this post:</div><div><br></div><div><a \
href="https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4D4B7966-8CFA-4457-B90C-6B272CABA644@gmail.com/#msg34883875" \
target="_blank">https://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4D4B7966-8CFA-4457-B90C-6B272CABA644%40gmail.com/#msg34883875</a></div><div><br></div><div>My \
SBCL version is 1.4.5.debian</div><div><br></div><div>I have a multithread process \
that runs unix-subprocesses. Every so often, it freezes. gdb shows one process \
to be </div><div>stuck in a malloc lock, after the fork() \
call.</div><div><br></div><div>The other threads are stuck like \
this</div><div><br></div><div>=========gdb output=========</div><div>#0 \
0x00007ffff79c26d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, \
expected=0, futex_word=0x7ffff41b8078) at \
../sysdeps/unix/sysv/linux/futex-internal.h:205</div><div><div>205 in \
../sysdeps/unix/sysv/linux/futex-internal.h</div><div>(gdb) backtrace</div><div>#0 \
0x00007ffff79c26d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, \
expected=0, futex_word=0x7ffff41b8078) at \
../sysdeps/unix/sysv/linux/futex-internal.h:205</div><div>#1 do_futex_wait \
(sem=sem@entry=0x7ffff41b8078, abstime=0x0) at sem_waitcommon.c:111</div><div>#2 \
0x00007ffff79c27c8 in __new_sem_wait_slow (sem=0x7ffff41b8078, abstime=0x0) at \
sem_waitcommon.c:181</div><div>#3 0x000000000041ee54 in os_sem_wait ()</div><div>#4 \
0x0000000000425373 in wait_for_thread_state_change ()</div><div>#5 \
0x000000000041c93d in sig_stop_for_gc_handler ()</div><div>#6 0x000000000041afc0 in \
?? ()</div><div>#7 <signal handler called></div><div>#8 syscall () at \
../sysdeps/unix/sysv/linux/x86_64/syscall.S:37</div><div>#9 0x0000000000427df0 in \
futex_wait ()</div><div>#10 0x0000000021d488f4 in ?? \
()</div></div><div>============================</div><div><br></div><div><br></div><div><br></div><div><br></div><div>Here's \
how I interpret it:</div><div><br></div><div>**1. Thread A enters common-lisp \
run-program, where it does (see code/run-program.lisp line \
833)</div><div><br></div><div>======== code/run-program.lisp line 833 \
=====</div><div><div>(without-gcing</div><div> (spawn progname \
args-vec</div><div><span \
class="m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-346777545821 \
5091258m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580Apple-tab-span" \
style="white-space:pre-wrap"> </span>stdin stdout stderr</div><div><span \
class="m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-346777545821 \
5091258m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580Apple-tab-span" \
style="white-space:pre-wrap"> </span>(if search 1 0)</div><div><span \
class="m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-346777545821 \
5091258m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580Apple-tab-span" \
style="white-space:pre-wrap"> </span>environment-vec pty-name</div><div><span \
class="m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-346777545821 \
5091258m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580Apple-tab-span" \
style="white-space:pre-wrap"> </span>(if wait 1 0) \
directory))</div></div><div>=====================================</div><div><br></div><div>**2. \
Thread B stars a garbage collection, sending the other threads a GC interrupt. \
</div><div><br></div><div>**3. Thread C happens to be holding a malloc lock, but it \
is interrupted by thread B, so is sitting idle waiting for the GC to \
finish.</div><div><br></div><div>**4. Thread A was sitting on the malloc lock too, \
but it isn't woken interrupted by thread B because it is inside without-gcing, \
</div><div><br></div><div>**4. Deadlock, as thread A waits forever for C to release \
the malloc lock, while preventing B and C from doing the GC that would</div><div> \
allow C to return and release the malloc \
lock.</div><div><br></div><div><br></div><div><br></div><div>It seems to me that \
without-gcing in run-program is the problem in this deadlock, as would any foreign \
function call that does a malloc</div><div>inside without-gcing \
</div><div><br></div><div>So my questions \
are:</div><div><br></div><div><br></div><div>1. Is the without-gcing around spawn \
in (defun run-program …) in code/run-program.lisp really necessary? I've taken it \
out in a patched version of </div><div> SBCL and it seems to help with this \
problem (though it is not a frequent problem, so I can't say if it is a \
cure).</div><div><br></div><div>2. Is it always a bad idea to mix spawn/fork with \
multiple threads in SBCL? Python groups suggested that this was a bad idea in \
general, yet</div><div> it's a feature I need.</div><div><br></div><div> 2a. \
If mixing spawn/fork with threads is a bad idea, then is there a better way to do \
fork, like having a single-threaded server process connected with a pipe</div><div> \
doing the forking and returning the results via a socket or file \
descriptor?</div><div><br></div><div>3. Is it always a bad idea to put (without-gcing \
..) around a foreign function call, because this malloc lockup is not unique to \
spawn? (and there</div><div> might be locks in C libraries besides \
malloc)</div><div><br></div><div>4. Also in (defun run-program …) in \
code/run-program.lisp , might it be better to put (with-active-processes-lock \
()</div><div> around only (push proc *active-processes*) and not around the big \
spawn block, so that child processes don't delay each other</div><div> other \
across threads?</div><div> (see <a \
href="https://github.com/sbcl/sbcl/blob/master/src/code/run-program.lisp" \
target="_blank">https://github.com/sbcl/sbcl/blob/master/src/code/run-program.lisp</a>, \
line 821 at present date)</div><div><br></div><div><br></div><div>Many thanks for any \
tips, and apologies if I said anything totally off the \
mark.</div><div><br></div><div>J.Klein</div><div><br></div><div><br></div><div><br></d \
iv><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><table \
class="m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-346777545821 \
5091258m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580js-file-line-container \
m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-3467775458215091258 \
m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580highlight \
m_-9130504671206925397m_842014716928636642m_-7203529591270067836m_-3467775458215091258 \
m_4218640444260607114m_4910263812254860667gmail-m_-6274647224449880577m_-2008284547545891580tab-size" \
style="box-sizing:border-box;border-collapse:collapse;border-spacing:0px;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe \
UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI \
Emoji","Segoe UI \
Symbol";font-size:14px;background-color:rgb(255,255,255)"><tbody \
style="box-sizing:border-box"></tbody></table></div></div>_______________________________________________<br>
Sbcl-devel mailing list<br>
<a href="mailto:Sbcl-devel@lists.sourceforge.net" \
target="_blank">Sbcl-devel@lists.sourceforge.net</a><br> <a \
href="https://lists.sourceforge.net/lists/listinfo/sbcl-devel" rel="noreferrer" \
target="_blank">https://lists.sourceforge.net/lists/listinfo/sbcl-devel</a><br> \
</blockquote></div> _______________________________________________<br>
Sbcl-devel mailing list<br>
<a href="mailto:Sbcl-devel@lists.sourceforge.net" \
target="_blank">Sbcl-devel@lists.sourceforge.net</a><br> <a \
href="https://lists.sourceforge.net/lists/listinfo/sbcl-devel" rel="noreferrer" \
target="_blank">https://lists.sourceforge.net/lists/listinfo/sbcl-devel</a><br> \
</blockquote></div></div></div></div></blockquote></div></div> </blockquote></div>
</blockquote></div></div></blockquote></div>
_______________________________________________
Sbcl-devel mailing list
Sbcl-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic