[prev in list] [next in list] [prev in thread] [next in thread] 

List:       gluster-users
Subject:    [Gluster-users] Gluster-users Digest, Vol 85, Issue 22 - 9. Re: seq read performance comparion betwe
From:       bengland () redhat ! com (Ben England)
Date:       2015-05-29 15:40:36
Message-ID: 2084699018.6049139.1432914036345.JavaMail.zimbra () redhat ! com
[Download RAW message or body]

Paul, I don't check this list every day, I would expect you can get more than half of \
minimum of network line speed or storage block device speed using a single libgfapi \
sequential read thread.  I did not see any throughput calculation or file size in \
your e-mail.

HTH, inline below...

-ben e

----- Original Message -----
> From: gluster-users-request at gluster.org
> To: gluster-users at gluster.org
> Sent: Friday, May 22, 2015 8:00:02 AM
> Subject: Gluster-users Digest, Vol 85, Issue 22
> 
> Message: 8
> Date: Fri, 22 May 2015 18:50:40 +0800
> From: Paul Guo <bigpaulguo at foxmail.com>
> To: gluster-users at gluster.org
> Subject: [Gluster-users] seq read performance comparion between
> 	libgfapi and	fuse
> Message-ID: <555F0A00.2060104 at foxmail.com>
> Content-Type: text/plain; charset=gbk; format=flowed
> 
> Hello,
> 
> I wrote two simple single-process seq read test case to compare libgfapi
> and fuse. The logic looks like this.
> 
> char buf[32768];
> while (1) {
> cnt = read(fd, buf, sizeof(buf));
> if (cnt == 0)
> break;
> else if (cnt > 0)
> total += cnt;
> // No "cnt < 0" was found during testing.
> }
> 
> Following is the time which is needed to finish reading a large file.
> 
> fuse         libgfapi
> direct io:         40s          51s
> non direct io:     40s          47s
> 
> The version is 3.6.3 on centos6.5. The result shows that libgfapi is
> obviously slower than the fuse interface although the cpu cycles were
> saved a lot during libgfapi testing. Each test was run before cleaning
> up all kernel pageche&inode&dentry caches and stopping and then starting
> glusterd&gluster (to clean up gluster cache).

so if you use libgfapi in a single-threaded app, you may need to tune gluster volume \
parameter read-ahead-page-count (defaults to 4).  The default is intended to \
trade-off single-thread performance for better aggregate performance and response \
time.  Here is a example of how to tune it for a single-thread use case, don't do \
this all the time. 

gluster volume set your-volume performance.read-ahead-page-count 16

As a debugging tool, you can try disabling readahead translator altogether 

# gluster v set your-volume read-ahead off

To reset parameters to defaults:

# gluster v set your-volume read-ahead
# gluster v set your-volume read-ahead-page-count

I have a benchmark for libgfapi testing in case this is useful to you:

https://github.com/bengland2/parallel-libgfapi

please e-mail me direct if problems with it.

> 
> I tested direct io because I suspected that fuse kernel readahead
> helped more than the read optimization solutions in gluster. I searched
> a lot but I did not find much about the comparison between fuse and
> libgfapi. Anyone has known about this and known why?
> 

If you use O_DIRECT you may be  bypassing readahead translator in Gluster and this \
may account for your problem.  Try NOT using O_DIRECT, and try above tuning.  Or if \
you really need O_DIRECT on client, try this command, which disables O_DIRECT on the \
server side but not the client, it's equivalent of NFS behavior.

# gluster v set your-volume network.remote-dio on

Also try turning off io-cache translator which will not help you here.

# gluster v set your-volume io-cache off

Also, O_DIRECT is passed all the way to the server by Gluster so your disk reads will \
ALSO use O_DIRECT, this is terrible for performance.  You want to have block device \
readahead when doing this test.  Suggest you set it to at least 4096 KB for block \
devices used for Gluster brick mountpoints.


[prev in list] [next in list] [prev in thread] [next in thread]