[prev in list] [next in list] [prev in thread] [next in thread]
List: veritas-vx
Subject: Re: [Veritas-vx] high latency with initial reads on qio files
From: Jeff_Jacobs () aoncons ! com
Date: 2001-05-18 18:28:31
[Download RAW message or body]
George,
Sorry it took me so long to get back to you.
What I'm doing here is forcing a bypass around FS cache by using the
following mount options:
noqio,mincache=direct,convosync=direct
You can also accomplish this on UFS (Solaris 2.6 and above) by using
this mount option:
forcedirectio
If you try either or all of these options, could you please let me know how
the results compare.
Historically, this has provided the best performance with Veritas file
system based Oracle databases.
Also, there is a very good document: "Sun/Oracle Best Practices" authored by
Bob Sneed that may help as well. You can find it at:
http://www.sun.com/blueprints/0101/SunOracle.pdf.
Best Regards
jj
Jeff Jacobs
Assistant Vice Persident of IT/Sr. UNIX Engineer
Aon Consulting
George Schlossnagle <george@omniti.com> on 05/17/2001 03:15:14 PM
To: Jeff Jacobs/IT/Aon Consulting@AonNA
cc: veritas-vx@Eng.Auburn.EDU
Subject: Re: [Veritas-vx] high latency with initial reads on qio files [Virus
Checked]
No mount options, filesystems all created as:
mkfs -F vxfs -o
ninode=unlimited,bsize=4096,version=4,inosize=256,logsize=256,largefiles
/dev/vx/rdsk/data 103587840
or bsize=8192, depending on the box.
I have a box that I can test weird mount options on, if you'd like to
see different statistics (I'd also be interested if anyone selse does
the same benchmark and gets drastically different results.)
George
On Thursday, May 17, 2001, at 02:49 PM, Jeff_Jacobs@aoncons.com wrote:
>
>
> Hello George,
> Just out of curiosity, what are the mount options used on the file
> systems in
> question? Also, how were the file systems created? Run the following:
> mkfs -m
> /dev/vx/rdsk/VOLUME_NAME
>
> jj
>
>
>
>
>
> George Schlossnagle <george@omniti.com> on 05/17/2001 12:12:32 PM
>
> To: veritas-vx@eng.auburn.edu
> cc: (bcc: Jeff Jacobs/IT/Aon Consulting)
>
> Subject: [Veritas-vx] high latency with initial reads on qio files
> [Virus
> Checked]
>
>
>
> apologies if this is a repeat of an earlier mail.
>
> I've been looking at a problem that myself and a number of dbas on a
> mailing list I participate in are having. Basically we are seeing a
> large number of 'file open' wait events, which occur when an oracle
> shadow processinitially opens and reads the first datablock of a
> datafile (usually when that shadow process is first started). Another
> dba noted these only appeared for him after he turned on quickio files.
> I did some benchmarks - basically doing opens and an 8k read off of
> various file types and got the following siturbing results:
>
> non-cached qio:
> avg 1100 micro-s per open+read (and a large variance- some over
> 7000!!!!)
>
> cached qio:
> avg 105 micro-s per open+read
>
> non-qio (on same vxfs fs):
> 65 micro-s per open+read
>
> using the O_DSYNC flag seems to make no difference. If you remove the
> read call and just do the open, the non-cached open time drops to
> 100ms. I can only assume that this is related to the qio file
> completely bypassing the fs cache, but I think this is very disturbing.
>
> I'm attaching the code I used to generate this. I ran it on multiple
> instances of Solaris 2.6/vxfs 3.3.3 and got consistent results.
>
> Any thoughts on the cause of that sever latency for non cached qio
> files?
>
> Thanks!
>
> George
> <Attachment missing>
>
> apologies if this is a repeat of an earlier mail.
>
> I've been looking at a problem that myself and a number of dbas on a
> mailing list I participate in are having. Basically we are seeing a
> large number of 'file open' wait events, which occur when an oracle
> shadow processinitially opens and reads the first datablock of a
> datafile (usually when that shadow process is first started). Another
> dba noted these only appeared for him after he turned on quickio files.
> I did some benchmarks - basically doing opens and an 8k read off of
> various file types and got the following siturbing results:
>
> non-cached qio:
> avg 1100 micro-s per open+read (and a large variance- some over
> 7000!!!!)
>
> cached qio:
> avg 105 micro-s per open+read
>
> non-qio (on same vxfs fs):
> 65 micro-s per open+read
>
> using the O_DSYNC flag seems to make no difference. If you remove the
> read call and just do the open, the non-cached open time drops to
> 100ms. I can only assume that this is related to the qio file
> completely bypassing the fs cache, but I think this is very disturbing.
>
> I'm attaching the code I used to generate this. I ran it on multiple
> instances of Solaris 2.6/vxfs 3.3.3 and got consistent results.
>
> Any thoughts on the cause of that sever latency for non cached qio
> files?
>
> Thanks!
>
> George
>
<Attachment missing>
No mount options, filesystems all created as:
mkfs -F vxfs -o
ninode=unlimited,bsize=4096,version=4,inosize=256,logsize=256,largefiles
/dev/vx/rdsk/data 103587840
or bsize=8192, depending on the box.
I have a box that I can test weird mount options on, if you'd like to
see different statistics (I'd also be interested if anyone selse does
the same benchmark and gets drastically different results.)
George
On Thursday, May 17, 2001, at 02:49 PM, Jeff_Jacobs@aoncons.com wrote:
>
>
> Hello George,
> Just out of curiosity, what are the mount options used on the file
> systems in
> question? Also, how were the file systems created? Run the following:
> mkfs -m
> /dev/vx/rdsk/VOLUME_NAME
>
> jj
>
>
>
>
>
> George Schlossnagle <george@omniti.com> on 05/17/2001 12:12:32 PM
>
> To: veritas-vx@eng.auburn.edu
> cc: (bcc: Jeff Jacobs/IT/Aon Consulting)
>
> Subject: [Veritas-vx] high latency with initial reads on qio files
> [Virus
> Checked]
>
>
>
> apologies if this is a repeat of an earlier mail.
>
> I've been looking at a problem that myself and a number of dbas on a
> mailing list I participate in are having. Basically we are seeing a
> large number of 'file open' wait events, which occur when an oracle
> shadow processinitially opens and reads the first datablock of a
> datafile (usually when that shadow process is first started). Another
> dba noted these only appeared for him after he turned on quickio files.
> I did some benchmarks - basically doing opens and an 8k read off of
> various file types and got the following siturbing results:
>
> non-cached qio:
> avg 1100 micro-s per open+read (and a large variance- some over
> 7000!!!!)
>
> cached qio:
> avg 105 micro-s per open+read
>
> non-qio (on same vxfs fs):
> 65 micro-s per open+read
>
> using the O_DSYNC flag seems to make no difference. If you remove the
> read call and just do the open, the non-cached open time drops to
> 100ms. I can only assume that this is related to the qio file
> completely bypassing the fs cache, but I think this is very disturbing.
>
> I'm attaching the code I used to generate this. I ran it on multiple
> instances of Solaris 2.6/vxfs 3.3.3 and got consistent results.
>
> Any thoughts on the cause of that sever latency for non cached qio
> files?
>
> Thanks!
>
> George
> <Attachment missing>
>
> apologies if this is a repeat of an earlier mail.
>
> I've been looking at a problem that myself and a number of dbas on a
> mailing list I participate in are having. Basically we are seeing a
> large number of 'file open' wait events, which occur when an oracle
> shadow processinitially opens and reads the first datablock of a
> datafile (usually when that shadow process is first started). Another
> dba noted these only appeared for him after he turned on quickio files.
> I did some benchmarks - basically doing opens and an 8k read off of
> various file types and got the following siturbing results:
>
> non-cached qio:
> avg 1100 micro-s per open+read (and a large variance- some over
> 7000!!!!)
>
> cached qio:
> avg 105 micro-s per open+read
>
> non-qio (on same vxfs fs):
> 65 micro-s per open+read
>
> using the O_DSYNC flag seems to make no difference. If you remove the
> read call and just do the open, the non-cached open time drops to
> 100ms. I can only assume that this is related to the qio file
> completely bypassing the fs cache, but I think this is very disturbing.
>
> I'm attaching the code I used to generate this. I ran it on multiple
> instances of Solaris 2.6/vxfs 3.3.3 and got consistent results.
>
> Any thoughts on the cause of that sever latency for non cached qio
> files?
>
> Thanks!
>
> George
>
["file_open.c" (application/octet-stream)]
/*
file_open.c
performs a tight loop of open and N byte reads to simulate
an oracle process doing it's initial open and datafile segment header.
George Schlossnagle <george@omniti.com>
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <fcntl.h>
#include <errno.h>
extern char *optarg;
extern int optind, opterr, optopt;
extern int errno;
int main(int argc, char **argv) {
char* file = NULL;
char* buff;
int block_size = 8192;
int iterations = 1000;
int dsync = 0;
struct timeval start;
struct timeval end;
long difftime;
int i;
int getoption;
while((getoption = getopt(argc, argv, "b:i:f:d")) != -1) {
switch(getoption) {
case 'b':
block_size = atoi(optarg);
break;
case 'i':
iterations = atoi(optarg);
break;
case 'f':
file = optarg;
break;
case 'd':
dsync = 1;
break;
default:
1;
}
}
if( file == NULL ) {
printf("Usage: %s [-b <block_size>] [-i <iterations>] -f <filename> [-d (O_DSYNC)]\n");
return -1;
}
buff = (char *) malloc(sizeof(char) * block_size);
gettimeofday(&start, NULL);
for(i = 0; i < iterations; i++) {
int fd;
if(dsync) {
if ((fd = open(file, O_RDWR | O_DSYNC)) == -1) {
printf("Failed opening %s: %s\n", file, strerror(errno));
return -1;
}
}
else {
if ((fd = open(file, O_RDWR)) == -1) {
printf("Failed opening %s: %s\n", file, strerror(errno));
return -1;
}
}
if (block_size != 0 ) {
if ( read(fd, buff, block_size) != block_size ) {
printf("Failed full read of %s\n", file);
return -1;
}
}
}
gettimeofday(&end, NULL);
difftime = (end.tv_sec*1000000 + end.tv_usec - start.tv_sec*1000000 - start.tv_usec)/iterations;
printf("Microseconds per open: %d\n", difftime);
}
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic