--3V7upXqbjpZ4EhLz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi! On Tue, Aug 07, 2012 at 03:17:23PM +0200, Miklos Szeredi wrote: > Anand Avati writes: > > I think we really need multiple request queues (as many number of CPUs) which > > are processed by multiple user-space fuse threads which bind themselves to one > > CPU each and syscalls add their request to the queue belonging to the CPU they > > are happening in -- all of this working without any cross-cpu shared data. We > > could optionally have some starvation avoidance by making another CPU's fuse > > thread pick this CPU's requests if it is too idle and this CPU has too many > > requests, but it need not be an initial feature. If all this really improves > > performance to the same extent (and hence corroborates the theory), the extra > > complexity would be well worth it. > > Yeah, that would be an interesting experiment. I can try to predict the outcome, too: what would happen if you implemented this code is the following (cpu numbers are of course invented): - the userspace process would run on cpu 0 - it would access the fuse fs (i.e. stat) - the fuse kernel code would run on cpu 0 - it would wakeup a worker thread on cpu 0 - the worker thread would perform the request (i.e. getattr) - the worker thread would wake up the kernel code again - the userspace process would receive the data so far so good. But eventually the linux scheduler would decide that the userspace process is running on a cpu that is not very idle (since the fuse worker is running on cpu 0), so it would move the userspace process to cpu 1 (which is completely idle). Now, the same code would run again, but this time on cpu 1. Which would cause the linux scheduler to move the userspace process to another cpu again, since the userspace process would be running on the only cpu that is quite busy, cpu 1. And so on. Why do I say this? Because I tried a very simple version of the suggested changes, which I am attaching. My patch would simply move the fuse worker thread to the same cpu than the userspace process, and I observed: no improvement in performance, but the userspace process constantly shifting cpus (which is bad for performance). So I think there is one small missing piece of code, that somehow ensures that once the fuse worker and the userspace code are scheduled on the same cpu, the linux scheduler will not move the userspace process away. Cu... Stefan -- Stefan Westerfeld, Hamburg/Germany, http://space.twc.de/~stefan --3V7upXqbjpZ4EhLz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="my-cpu-fuse.diff" diff --git a/include/fuse_kernel.h b/include/fuse_kernel.h index c632b58..d8c472a 100644 --- a/include/fuse_kernel.h +++ b/include/fuse_kernel.h @@ -622,7 +622,7 @@ struct fuse_in_header { __u32 uid; __u32 gid; __u32 pid; - __u32 padding; + __u32 cpu; }; struct fuse_out_header { diff --git a/lib/fuse_lowlevel.c b/lib/fuse_lowlevel.c index a259f37..74e47aa 100644 --- a/lib/fuse_lowlevel.c +++ b/lib/fuse_lowlevel.c @@ -2377,6 +2377,11 @@ static void fuse_ll_process_buf(void *data, const struct fuse_buf *buf, req->ctx.pid = in->pid; req->ch = ch; + cpu_set_t cpus; + CPU_ZERO (&cpus); + CPU_SET (in->cpu, &cpus); + sched_setaffinity (0, sizeof (cpu_set_t), &cpus); + err = EIO; if (!f->got_init) { enum fuse_opcode expected; --3V7upXqbjpZ4EhLz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="my-cpu-kernel.diff" diff -ru linux-source-3.2.0/fs/fuse/dev.c ../linux-source-3.2.0/fs/fuse/dev.c --- linux-source-3.2.0/fs/fuse/dev.c 2012-01-05 00:55:44.000000000 +0100 +++ ../linux-source-3.2.0/fs/fuse/dev.c 2012-07-30 17:32:11.979330611 +0200 @@ -95,6 +95,7 @@ req->in.h.uid = current_fsuid(); req->in.h.gid = current_fsgid(); req->in.h.pid = current->pid; + req->in.h.cpu = raw_smp_processor_id(); } struct fuse_req *fuse_get_req(struct fuse_conn *fc) diff -ru linux-source-3.2.0/include/linux/fuse.h ../linux-source-3.2.0/include/linux/fuse.h --- linux-source-3.2.0/include/linux/fuse.h 2012-01-05 00:55:44.000000000 +0100 +++ ../linux-source-3.2.0/include/linux/fuse.h 2012-07-30 11:29:14.954851122 +0200 @@ -572,7 +572,7 @@ __u32 uid; __u32 gid; __u32 pid; - __u32 padding; + __u32 cpu; }; struct fuse_out_header { --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ fuse-devel mailing list fuse-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fuse-devel --3V7upXqbjpZ4EhLz--