[prev in list] [next in list] [prev in thread] [next in thread] 

List:       fuse-devel
Subject:    Re: [fuse-devel] Unkillable process due to outstanding FUSE requests
From:       David Bartley <andareed () gmail ! com>
Date:       2010-08-26 10:22:27
Message-ID: AANLkTinM9F7NjuT3agf9FSvnd9p3RTZANdHx6zouwt4+ () mail ! gmail ! com
[Download RAW message or body]

On Thu, Aug 26, 2010 at 2:14 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Thu, 26 Aug 2010, David Bartley wrote:
>> Forgot to mention, this is on Linux 2.6.32.
>>
>> On Thu, Aug 26, 2010 at 12:49 AM, David Bartley <andareed@gmail.com> wrote:
>> > Hi,
>> >
>> > I have a process that I'm unable to kill. Here's where it's stuck in the kernel:
>> >
>> > [<ffffffff812379cd>] fuse_get_req_nofail+0x6d/0x1c0
>> > [<ffffffff8123bb9b>] fuse_flush+0x8b/0x140
>> > [<ffffffff8114242c>] filp_close+0x3c/0x90
>> > [<ffffffff81142537>] sys_close+0xb7/0x120
>> > [<ffffffff81048dbf>] sysenter_dispatch+0x7/0x2e
>> > [<ffffffffffffffff>] 0xffffffffffffffff
>> >
>> > The FUSE daemon process itself no longer exists, so I would have
>> > expected the kernel module to abort all outstanding requests. I wrote
>> > a small kernel module to try to dump the state of the associated
>> > fuse_conn (see the bottom of my email). Since connected is 0, I would
>> > have expected all of these requests to be ended by the call to
>> > end_requests(&fc->pending) after connected is set to 0. I don't
>> > believe there's any way for new requests to be either added to
>> > fc->pending or fc->bg_queue when connected = 0.
>
> Great bug report, thanks!
>
> I think I see it: if bg_queue is not empty then flush_bg_queue()
> called from request_end() can put more requests on to the pending
> queue.  If this happens while ending requests on the processing queue
> then those background requests will be left on the pending queue and
> never ended.

Ah, that makes sense. I noticed that flush_bg_queue may add things to
fc->pending but concluded that the loop in end_requests(&fc->pending)
would ultimately remove them. But I completely ignored the next call
to end_requests(&fc->processing). Thanks for helping with this!

>
> This is not easy to trigger, the connection needs to close when there
> are background requests both on the bg_queue and the processing queue.
> This would explain why this bug wasn't previously reported.

I actually had modified my FUSE FS to try to reproduce this. I changed
the release() and close() implementations so that they randomly hung
50% of the time, then ran a bunch of cat processes on different files,
and finally killed -9 the daemon. Unfortunately I never wound up with
an unkillable cat process.

>
> I think just swithing the order of end_requests() for processing and
> pending would fix this, but for a more clean solution see the patch
> below.
>
> Thanks,
> Miklos
>
> ---
>  fs/fuse/dev.c |   16 ++++++++++++----
>  1 file changed, 12 insertions(+), 4 deletions(-)
>
> Index: linux-2.6/fs/fuse/dev.c
> ===================================================================
> --- linux-2.6.orig/fs/fuse/dev.c        2010-08-13 16:07:00.000000000 +0200
> +++ linux-2.6/fs/fuse/dev.c     2010-08-26 11:02:49.000000000 +0200
> @@ -1769,6 +1769,14 @@ __acquires(&fc->lock)
>        }
>  }
>
> +static void end_queued_requests(struct fuse_conn *fc)
> +{
> +       fc->max_background = UINT_MAX;
> +       flush_bg_queue(fc);
> +       end_requests(fc, &fc->pending);
> +       end_requests(fc, &fc->processing);
> +}
> +
>  /*
>  * Abort all requests.
>  *
> @@ -1795,8 +1803,7 @@ void fuse_abort_conn(struct fuse_conn *f
>                fc->connected = 0;
>                fc->blocked = 0;
>                end_io_requests(fc);
> -               end_requests(fc, &fc->pending);
> -               end_requests(fc, &fc->processing);
> +               end_queued_requests(fc);
>                wake_up_all(&fc->waitq);
>                wake_up_all(&fc->blocked_waitq);
>                kill_fasync(&fc->fasync, SIGIO, POLL_IN);
> @@ -1811,8 +1818,9 @@ int fuse_dev_release(struct inode *inode
>        if (fc) {
>                spin_lock(&fc->lock);
>                fc->connected = 0;
> -               end_requests(fc, &fc->pending);
> -               end_requests(fc, &fc->processing);
> +               fc->blocked = 0;
> +               end_queued_requests(fc);
> +               wake_up_all(&fc->blocked_waitq);
>                spin_unlock(&fc->lock);
>                fuse_conn_put(fc);
>        }
>

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
fuse-devel mailing list
fuse-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fuse-devel

[prev in list] [next in list] [prev in thread] [next in thread]