[prev in list] [next in list] [prev in thread] [next in thread]
List: gluster-devel
Subject: Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu
From: David Gossage <dgossage () carouselchecks ! com>
Date: 2016-07-26 20:09:24
Message-ID: CAJXeYEWNsR4X6kL+yWYQ9tu_HEZnhwq0VyVKUMcLFog-15ad8Q () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Has a bug report been filed for this issue or should l I create one with
the logs and results provided so far?
*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
On Fri, Jul 22, 2016 at 12:53 PM, David Gossage <dgossage@carouselchecks.com
> wrote:
>
>
>
> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur@redhat.com> wrote:
>
> > On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <samppah@neutraali.net>
> > wrote:
> > > Here is a quick way how to test this:
> > > GlusterFS 3.7.13 volume with default settings with brick on ZFS
> > dataset. gluster-test1 is server and gluster-test2 is client mounting with
> > FUSE.
> > >
> > > Writing file with oflag=direct is not ok:
> > > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
> > count=1 bs=1024000
> > > dd: failed to open ‘file': Invalid argument
> > >
> > > Enable network.remote-dio on Gluster Volume:
> > > [root@gluster-test1 gluster]# gluster volume set gluster
> > network.remote-dio enable
> > > volume set: success
> > >
> > > Writing small file with oflag=direct is ok:
> > > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
> > count=1 bs=1024000
> > > 1+0 records in
> > > 1+0 records out
> > > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
> > >
> > > Writing bigger file with oflag=direct is ok:
> > > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
> > count=100 bs=1M
> > > 100+0 records in
> > > 100+0 records out
> > > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
> > >
> > > Enable Sharding on Gluster Volume:
> > > [root@gluster-test1 gluster]# gluster volume set gluster
> > features.shard enable
> > > volume set: success
> > >
> > > Writing small file with oflag=direct is ok:
> > > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
> > count=1 bs=1M
> > > 1+0 records in
> > > 1+0 records out
> > > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
> > >
> > > Writing bigger file with oflag=direct is not ok:
> > > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
> > count=100 bs=1M
> > > dd: error writing ‘file3': Operation not permitted
> > > dd: closing output file ‘file3': Operation not permitted
> > >
> >
> >
> > Thank you for these tests! would it be possible to share the brick and
> > client logs?
> >
>
> Not sure if his tests are same as my setup but here is what I end up with
>
> Volume Name: glustershard
> Type: Replicate
> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.71.10:/gluster1/shard1/1
> Brick2: 192.168.71.11:/gluster1/shard2/1
> Brick3: 192.168.71.12:/gluster1/shard3/1
> Options Reconfigured:
> features.shard-block-size: 64MB
> features.shard: on
> server.allow-insecure: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.quick-read: off
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> performance.read-ahead: off
> performance.readdir-ahead: on
>
>
>
> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/
> oflag=direct count=100 bs=1M
> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__
> .trashcan/
> [root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/
> 192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
> oflag=direct count=100 bs=1M
> dd: error writing ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test':
> Operation not permitted
>
> creates the 64M file in expected location then the shard is 0
>
> # file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x000000000000000000000000
> trusted.bit-rot.version=0x0200000000000000579231f3000e16e7
> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0
> trusted.glusterfs.shard.block-size=0x0000000004000000
>
> trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000
>
>
> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x000000000000000000000000
> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80
>
>
> > Regards,
> > Vijay
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
[Attachment #5 (text/html)]
<div dir="ltr">Has a bug report been filed for this issue or should l I create one \
with the logs and results provided so far?</div><div class="gmail_extra"><br \
clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div \
dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David \
Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span \
style="color:rgb(51,51,51)"></span></i><font size="1"><b \
style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> \
| System Administrator</span></b></font></font><font \
style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b \
style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a \
value="+17086132426">708.613.2284<font color="#888888"><font \
size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br><div class="gmail_quote">On Fri, Jul 22, 2016 at 12:53 PM, David Gossage <span \
dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" \
target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br \
clear="all"><div><div data-smartmail="gmail_signature"><div \
dir="ltr"><br></div></div></div> <br><div class="gmail_quote">On Fri, Jul 22, 2016 at \
9:37 AM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" \
target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On \
Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <<a \
href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>> \
wrote:<br> > Here is a quick way how to test this:<br>
> GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. \
gluster-test1 is server and gluster-test2 is client mounting with FUSE.<br> ><br>
> Writing file with oflag=direct is not ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 \
bs=1024000<br> > dd: failed to open ‘file': Invalid argument<br>
><br>
> Enable network.remote-dio on Gluster Volume:<br>
> [root@gluster-test1 gluster]# gluster volume set gluster network.remote-dio \
enable<br> > volume set: success<br>
><br>
> Writing small file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 \
bs=1024000<br> > 1+0 records in<br>
> 1+0 records out<br>
> 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s<br>
><br>
> Writing bigger file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 \
bs=1M<br> > 100+0 records in<br>
> 100+0 records out<br>
> 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s<br>
><br>
> Enable Sharding on Gluster Volume:<br>
> [root@gluster-test1 gluster]# gluster volume set gluster features.shard \
enable<br> > volume set: success<br>
><br>
> Writing small file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 \
bs=1M<br> > 1+0 records in<br>
> 1+0 records out<br>
> 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s<br>
><br>
> Writing bigger file with oflag=direct is not ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 \
bs=1M<br> > dd: error writing ‘file3': Operation not permitted<br>
> dd: closing output file ‘file3': Operation not permitted<br>
><br>
<br>
<br>
Thank you for these tests! would it be possible to share the brick and<br>
client logs?<br></blockquote><div><br></div><div>Not sure if his tests are same as my \
setup but here is what I end up with</div><div><br></div><div><div>Volume Name: \
glustershard</div><div>Type: Replicate</div><div>Volume ID: \
0cc4efb6-3836-4caa-b24a-b3afb6e407c3</div><div>Status: Started</div><div>Number of \
Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: \
192.168.71.10:/gluster1/shard1/1</div><div>Brick2: \
192.168.71.11:/gluster1/shard2/1</div><div>Brick3: \
192.168.71.12:/gluster1/shard3/1</div><div>Options \
Reconfigured:</div><div>features.shard-block-size: 64MB</div><div>features.shard: \
on</div><div>server.allow-insecure: on</div><div>storage.owner-uid: \
36</div><div>storage.owner-gid: 36</div><div>cluster.server-quorum-type: \
server</div><div>cluster.quorum-type: auto</div><div>network.remote-dio: \
enable</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: \
off</div><div>performance.io-cache: off</div><div>performance.quick-read: \
off</div><div>cluster.self-heal-window-size: \
1024</div><div>cluster.background-self-heal-count: 16</div><div>nfs.enable-ino32: \
off</div><div>nfs.addr-namelookup: off</div><div>nfs.disable: \
on</div><div>performance.read-ahead: off</div><div>performance.readdir-ahead: \
on</div></div><div><br></div><div><br></div><div><br></div><div> dd if=/dev/zero \
of=/rhev/data-center/mnt/glusterSD/<a href="http://192.168.71.11" \
target="_blank">192.168.71.11</a>\:_glustershard/ oflag=direct count=100 \
bs=1M</div><div>81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ \
.trashcan/ </div><div>[root@ccengine2 ~]# dd \
if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://192.168.71.11" \
target="_blank">192.168.71.11</a>\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test \
oflag=direct count=100 bs=1M</div><div>dd: error writing \
‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test': \
Operation not permitted</div><div><br></div><div>creates the 64M file in expected \
location then the shard is 0</div><div><br></div><div><div># file: \
gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test</div><div>security. \
selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div \
>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.bit-rot.version=0x0200 \
> 000000000000579231f3000e16e7</div><div>trusted.gfid=0xec6de302b35f427985639ca3e25d9d \
> f0</div><div>trusted.glusterfs.shard.block-size=0x0000000004000000</div><div>trusted \
> .glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000</div></div><div><br></div><div><br></div><div><div># \
> file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1</div><div>secu \
> rity.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</d \
> iv><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80</div></div><div><br></div><blockquote \
> class="gmail_quote" style="margin:0px 0px 0px \
> 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
>
<br>
Regards,<br>
Vijay<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" \
target="_blank">Gluster-users@gluster.org</a><br> <a \
href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" \
target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br></div></div>
</blockquote></div><br></div>
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic