[prev in list] [next in list] [prev in thread] [next in thread] 

List:       fuse-devel
Subject:    Re: [fuse-devel] stat performance on multicore systems: measurements
From:       Stefan Westerfeld <stefan () space ! twc ! de>
Date:       2012-08-07 16:29:55
Message-ID: 20120807162955.GA28081 () space ! twc ! de
[Download RAW message or body]

Hi!

On Tue, Aug 07, 2012 at 03:17:23PM +0200, Miklos Szeredi wrote:
> Anand Avati <anand.avati@gmail.com> writes:
> > I think we really need multiple request queues (as many number of CPUs) which
> > are processed by multiple user-space fuse threads which bind themselves to one
> > CPU each and syscalls add their request to the queue belonging to the CPU they
> > are happening in -- all of this working without any cross-cpu shared data. We
> > could optionally have some starvation avoidance by making another CPU's fuse
> > thread pick this CPU's requests if it is too idle and this CPU has too many
> > requests, but it need not be an initial feature. If all this really improves
> > performance to the same extent (and hence corroborates the theory), the extra
> > complexity would be well worth it.
> 
> Yeah, that would be an interesting experiment.

I can try to predict the outcome, too: what would happen if you implemented
this code is the following (cpu numbers are of course invented):

- the userspace process would run on cpu 0
- it would access the fuse fs (i.e. stat)
- the fuse kernel code would run on cpu 0
- it would wakeup a worker thread on cpu 0
- the worker thread would perform the request (i.e. getattr)
- the worker thread would wake up the kernel code again
- the userspace process would receive the data

so far so good. But eventually the linux scheduler would decide that the
userspace process is running on a cpu that is not very idle (since the fuse
worker is running on cpu 0), so it would move the userspace process to cpu 1
(which is completely idle). Now, the same code would run again, but this time
on cpu 1.

Which would cause the linux scheduler to move the userspace process to another
cpu again, since the userspace process would be running on the only cpu that is
quite busy, cpu 1.

And so on.

Why do I say this? Because I tried a very simple version of the suggested changes,
which I am attaching. My patch would simply move the fuse worker thread to the
same cpu than the userspace process, and I observed: no improvement in performance,
but the userspace process constantly shifting cpus (which is bad for performance).

So I think there is one small missing piece of code, that somehow ensures that
once the fuse worker and the userspace code are scheduled on the same cpu, the
linux scheduler will not move the userspace process away.

   Cu... Stefan
-- 
Stefan Westerfeld, Hamburg/Germany, http://space.twc.de/~stefan

["my-cpu-fuse.diff" (text/x-diff)]

diff --git a/include/fuse_kernel.h b/include/fuse_kernel.h
index c632b58..d8c472a 100644
--- a/include/fuse_kernel.h
+++ b/include/fuse_kernel.h
@@ -622,7 +622,7 @@ struct fuse_in_header {
 	__u32	uid;
 	__u32	gid;
 	__u32	pid;
-	__u32	padding;
+	__u32	cpu;
 };
 
 struct fuse_out_header {
diff --git a/lib/fuse_lowlevel.c b/lib/fuse_lowlevel.c
index a259f37..74e47aa 100644
--- a/lib/fuse_lowlevel.c
+++ b/lib/fuse_lowlevel.c
@@ -2377,6 +2377,11 @@ static void fuse_ll_process_buf(void *data, const struct fuse_buf *buf,
 	req->ctx.pid = in->pid;
 	req->ch = ch;
 
+        cpu_set_t cpus;
+        CPU_ZERO (&cpus);
+        CPU_SET (in->cpu, &cpus);
+        sched_setaffinity (0, sizeof (cpu_set_t), &cpus);
+
 	err = EIO;
 	if (!f->got_init) {
 		enum fuse_opcode expected;

["my-cpu-kernel.diff" (text/x-diff)]

diff -ru linux-source-3.2.0/fs/fuse/dev.c ../linux-source-3.2.0/fs/fuse/dev.c
--- linux-source-3.2.0/fs/fuse/dev.c	2012-01-05 00:55:44.000000000 +0100
+++ ../linux-source-3.2.0/fs/fuse/dev.c	2012-07-30 17:32:11.979330611 +0200
@@ -95,6 +95,7 @@
 	req->in.h.uid = current_fsuid();
 	req->in.h.gid = current_fsgid();
 	req->in.h.pid = current->pid;
+        req->in.h.cpu = raw_smp_processor_id();
 }
 
 struct fuse_req *fuse_get_req(struct fuse_conn *fc)
diff -ru linux-source-3.2.0/include/linux/fuse.h ../linux-source-3.2.0/include/linux/fuse.h
--- linux-source-3.2.0/include/linux/fuse.h	2012-01-05 00:55:44.000000000 +0100
+++ ../linux-source-3.2.0/include/linux/fuse.h	2012-07-30 11:29:14.954851122 +0200
@@ -572,7 +572,7 @@
 	__u32	uid;
 	__u32	gid;
 	__u32	pid;
-	__u32	padding;
+	__u32	cpu;
 };
 
 struct fuse_out_header {


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
fuse-devel mailing list
fuse-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fuse-devel


[prev in list] [next in list] [prev in thread] [next in thread]