[prev in list] [next in list] [prev in thread] [next in thread]
List: gluster-users
Subject: Re: [Gluster-users] General questions
From: Krutika Dhananjay <kdhananj () redhat ! com>
Date: 2019-06-21 7:52:30
Message-ID: CAPhYV8ORvJMy-_OsNrRFi6+FpObA=gEdKTBpk2zU2utjYR_FJg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Adding (back) gluster-users.
-Krutika
On Fri, Jun 21, 2019 at 1:09 PM Krutika Dhananjay <kdhananj@redhat.com>
wrote:
>
>
> On Fri, Jun 21, 2019 at 12:43 PM Cristian Del Carlo <
> cristian.delcarlo@targetsolutions.it> wrote:
>
>> Thanks Strahil,
>>
>> in this link
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.=
4/html/administration_guide/sect-creating_replicated_volumes
>> i see:
>>
>>
>> *Sharding has one supported use case: in the context of providing Red Ha=
t
>> Gluster Storage as a storage domain for Red Hat Enterprise Virtualizatio=
n,
>> to provide storage for live virtual machine images. Note that sharding i=
s
>> also a requirement for this use case, as it provides significant
>> performance improvements over previous implementations. *
>>
>> The deafult setting in GusterFS 6.1 appears to be:
>>
>> features.shard-block-size 64MB
>>
>> features.shard-lru-limit 16384
>>
>> features.shard-deletion-rate 100
>>
>
> That's right. Based on the tests we'd conducted internally, we'd found
> 64MB to be a good number both in terms of self-heal and IO performance. 4=
MB
> is a little on the lower side in that sense. The benefits of some feature=
s
> like eager-locking are lost if the shard size is too small. You can perha=
ps
> run some tests with 64MB shard-block-size to begin with, and tune it if i=
t
> doesn't fit your needs.
>
> -Krutika
>
>
>> Bricks in my case are over an xfs filesystem. I'll try different
>> block-size but if I understand correctly, small block sizes are preferab=
le
>> to big block sizes and If i have doubt I will put 4M.
>>
>> Very thanks for the warning, message received! :-)
>>
>> Best Regards,
>>
>> Cristian
>>
>>
>> Il giorno gio 20 giu 2019 alle ore 22:13 Strahil Nikolov <
>> hunter86_bg@yahoo.com> ha scritto:
>>
>>> Sharding is complex. It helps to heal faster -as only the shards that
>>> got changed will be replicated, but imagine a 1GB shard that got only 5=
12k
>>> updated - in such case you will copy the whole shard to the other repli=
cas.
>>> RHV & oVirt use a default shard size of 4M which is the exact size of
>>> the default PE in LVM.
>>>
>>> On the other side, it speeds stuff as gluster can balance the shards
>>> properly on the replicas and thus you can evenly distribute the load on=
the
>>> cluster.
>>> It is not a coincidence that RHV and oVirt use sharding by default.
>>>
>>> Just a warning.
>>> NEVER, EVER, DISABLE SHARDING!!! ONCE ENABLED - STAYS ENABLED!
>>> Don't ask how I learnGrazie dell'avviso, messaggio ricevuto!t that :)
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>>
>>>
>>> =D0=92 =D1=87=D0=B5=D1=82=D0=B2=D1=8A=D1=80=D1=82=D1=8A=D0=BA, 20 =D1=
=8E=D0=BD=D0=B8 2019 =D0=B3., 18:32:00 =D1=87. =D0=93=D1=80=D0=B8=D0=BD=D1=
=83=D0=B8=D1=87+3, Cristian Del Carlo <
>>> cristian.delcarlo@targetsolutions.it> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=
=B0:
>>>
>>>
>>> Hi,
>>>
>>> thanks for your help.
>>>
>>> I am planing to use libvirtd with plain KVM.
>>>
>>> Ok i will use libgfapi.
>>>
>>> I'm confused about the use of sharding is it useful in this
>>> configuration? Doesn't sharding help limit the bandwidth in the event o=
f a
>>> rebalancing?
>>>
>>> In the vm setting so i need to use directsync to avoid corruption.
>>>
>>> Thanks again,
>>>
>>> Il giorno gio 20 giu 2019 alle ore 12:25 Strahil <hunter86_bg@yahoo.com=
>
>>> ha scritto:
>>>
>>> Hi,
>>>
>>> Are you planing to use oVirt or plain KVM or openstack?
>>>
>>> I would recommend you to use gluster v6.1 as it is the latest stable
>>> version and will have longer support than the older versions.
>>>
>>> Fuse vs libgfapi - use the latter as it has better performance and less
>>> overhead on the host.oVirt does supports both libgfapi and fuse.
>>>
>>> Also, use replica 3 because you will have better read performance
>>> compared to replica 2 arbiter 1.
>>>
>>> Sharding is a tradeoff between CPU (when there is no sharding , gluste=
r
>>> shd must calculate the offset of the VM disk) and bandwidth (whole shar=
d
>>> is being replicated despite even 512 need to be synced).
>>>
>>> If you will do live migration - you do not want to cache in order to
>>> avoid corruption.
>>> Thus oVirt is using direct I/O.
>>> Still, you can check the gluster settings mentioned in Red Hat
>>> documentation for Virt/openStack .
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>> On Jun 20, 2019 13:12, Cristian Del Carlo <
>>> cristian.delcarlo@targetsolutions.it> wrote:
>>>
>>> Hi,
>>>
>>> I'm testing glusterfs before using it in production, it should be used
>>> to store vm for nodes with libvirtd.
>>>
>>> In production I will have 4 nodes connected with a dedicated 20gbit/s
>>> network.
>>>
>>> Which version to use in production on a centos 7.x? Should I use Gluste=
r
>>> version 6?
>>>
>>> To make the volume available to libvirtd the best method is to use FUSE=
?
>>>
>>> I see that stripped is deprecated. Is it reasonable to use the volume
>>> with 3 replicas on 4 nodes and sharding enabled?
>>> Is there convenience to use sharding volume in this context? I think
>>> could positive inpact in read performance or rebalance. Is it true?
>>>
>>> In the vm configuration I use the virtio disk. How is it better to set
>>> the disk cache to get the best performances none, default or writeback?
>>>
>>> Thanks in advance for your patience and answers.
>>>
>>> Thanks,
>>>
>>>
>>> *Cristian Del Carlo*
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *Cristian Del Carlo*
>>>
>>> *Target Solutions s.r.l.*
>>>
>>> *T* +39 0583 1905621
>>> *F* +39 0583 1905675
>>> *@* cristian.delcarlo@targetsolutions.it
>>>
>>> http://www.targetsolutions.it
>>> P.IVA e C.Fiscale: 01815270465 Reg. Imp. di Lucca
>>> Capitale Sociale: =E2=82=AC11.000,00 iv - REA n=C2=B0 173227
>>>
>>> Il testo e gli eventuali documenti trasmessi contengono informazioni
>>> riservate al destinatario indicato. La seguente e-mail e' confidenziale=
e
>>> la sua riservatezza e' tutelata legalmente dal Decreto Legislativo 196 =
del
>>> 30/06/2003 (Codice di tutela della privacy). La lettura, copia o altro =
uso
>>> non autorizzato o qualsiasi altra azione derivante dalla conoscenza di
>>> queste informazioni sono rigorosamente vietate. Qualora abbiate ricevut=
o
>>> questo documento per errore siete cortesemente pregati di darne immedia=
ta
>>> comunicazione al mittente e di provvedere, immediatamente, alla sua
>>> distruzione.
>>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div>Adding (back) \
gluster-users.</div><div><br></div><div>-Krutika<br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 21, 2019 at 1:09 PM \
Krutika Dhananjay <<a \
href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div \
dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On \
Fri, Jun 21, 2019 at 12:43 PM Cristian Del Carlo <<a \
href="mailto:cristian.delcarlo@targetsolutions.it" \
target="_blank">cristian.delcarlo@targetsolutions.it</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div \
dir="ltr"><div>Thanks Strahil,</div><div><br></div><div>in this link <a \
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-creating_replicated_volumes" \
target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-creating_replicated_volumes</a> \
i see:</div><div><br></div><div><i>Sharding has one supported use case: in the \
context of providing Red Hat Gluster Storage as a storage domain for Red Hat \
Enterprise Virtualization, to provide storage for live virtual machine images. Note
that sharding is also a requirement for this use case, as it provides
significant performance improvements over previous implementations. \
<br></i></div><div><i><br></i></div><div>The deafult setting in GusterFS 6.1 appears \
to be:</div> \
<br><div>features.shard-block-size 64MB \
<br>features.shard-lru-limit 16384 \
<br>features.shard-deletion-rate 100 \
<br></div></div></div></blockquote><div><br></div><div>That's right. Based on the \
tests we'd conducted internally, we'd found 64MB to be a good number both in \
terms of self-heal and IO performance. 4MB is a little on the lower side in that \
sense. The benefits of some features like eager-locking are lost if the shard size is \
too small. You can perhaps run some tests with 64MB shard-block-size to begin with, \
and tune it if it doesn't fit your \
needs.</div><div><br></div><div>-Krutika</div><div><br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div \
dir="ltr"><div></div><div><br></div><div>Bricks in my case are over an xfs \
filesystem. I'll try different block-size but if I understand correctly, small \
block sizes are preferable to big block sizes and If i have doubt I will put \
4M.</div><div><br></div><div>Very thanks for the warning, message received! \
:-)</div><div><br></div><div>Best \
Regards,</div><div><br></div><div>Cristian<br></div><div><br></div><div></div><div><i></i></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 20 giu 2019 alle \
ore 22:13 Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" \
target="_blank">hunter86_bg@yahoo.com</a>> ha scritto:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div><div \
class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp678665a5yahoo-style-wrap" \
style="font-family:courier \
new,courier,monaco,monospace,sans-serif;font-size:16px"><div></div> <div>Sharding is \
complex. It helps to heal faster -as only the shards that got changed will be \
replicated, but imagine a 1GB shard that got only 512k updated - in such case you \
will copy the whole shard to the other replicas.</div><div>RHV & oVirt use a \
default shard size of 4M which is the exact size of the default PE in \
LVM.</div><div><br></div><div>On the other side, it speeds stuff as gluster can \
balance the shards properly on the replicas and thus you can evenly distribute the \
load on the cluster.</div><div>It is not a coincidence that RHV and oVirt use \
sharding by default.</div><div><br></div><div>Just a warning.</div><div>NEVER, EVER, \
DISABLE SHARDING!!! ONCE ENABLED - STAYS ENABLED!</div><div>Don't ask how I \
learnGrazie dell'avviso, messaggio ricevuto!t that \
:)</div><div><br></div><div>Best Regards,</div><div>Strahil \
Nikolov</div><div><br></div><div><br></div><div><br></div>
</div><div id="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yahoo_quoted_1362385886" \
class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yahoo_quoted">
<div style="font-family:"Helvetica \
Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
В четвъртък, 20 юни 2019 г., 18:32:00 ч. \
Гринуич+3, Cristian Del Carlo <<a \
href="mailto:cristian.delcarlo@targetsolutions.it" \
target="_blank">cristian.delcarlo@targetsolutions.it</a>> написа: </div>
<div><br></div>
<div><br></div>
<div><div \
id="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799"><div><div \
dir="ltr"><div>Hi,</div><div><br clear="none"></div><div>thanks for your \
help.</div><div><br clear="none"></div><div>I am planing to use libvirtd with plain \
KVM.</div><div><br clear="none"></div><div>Ok i will use libgfapi. <br \
clear="none"></div><div><br clear="none"></div><div>I'm confused about the use of \
sharding is it useful in this configuration? Doesn't sharding help limit the \
bandwidth in the event of a rebalancing?</div><div><br clear="none"></div><div>In the \
vm setting so i need to use directsync to avoid corruption.</div><div><br \
clear="none"></div><div>Thanks again,<br clear="none"> </div></div><br \
clear="none"><div class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799gmail_quote"><div \
class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799gmail_attr" \
dir="ltr">Il giorno gio 20 giu 2019 alle ore 12:25 Strahil <<a shape="rect" \
href="mailto:hunter86_bg@yahoo.com" rel="nofollow" \
target="_blank">hunter86_bg@yahoo.com</a>> ha scritto:<br clear="none"></div><div \
class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799yqt4291059422" \
id="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799yqt39338"><blockquote \
class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><p dir="ltr">Hi,</p> <p dir="ltr">Are you planing \
to use oVirt or plain KVM or openstack?</p> <p dir="ltr">I would recommend you to use \
gluster v6.1 as it is the latest stable version and will have longer support than the \
older versions.</p> <p dir="ltr">Fuse vs libgfapi - use the latter as it has better \
performance and less overhead on the host.oVirt does supports both libgfapi and \
fuse.</p> <p dir="ltr">Also, use replica 3 because you will have better read \
performance compared to replica 2 arbiter 1.</p> <p dir="ltr">Sharding is a tradeoff \
between CPU (when there is no sharding , gluster shd must calculate the offset of the \
VM disk) and bandwidth (whole shard is being replicated despite even 512 need to be \
synced).</p> <p dir="ltr">If you will do live migration - you do not want to cache \
in order to avoid corruption.<br clear="none"> Thus oVirt is using direct I/O.<br \
clear="none"> Still, you can check the gluster settings mentioned in Red Hat \
documentation for Virt/openStack .</p> <p dir="ltr">Best Regards,<br clear="none">
Strahil Nikolov</p>
<div class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799gmail-m_1444675035890430144quote">On \
Jun 20, 2019 13:12, Cristian Del Carlo <<a shape="rect" \
href="mailto:cristian.delcarlo@targetsolutions.it" rel="nofollow" \
target="_blank">cristian.delcarlo@targetsolutions.it</a>> wrote:<br \
clear="none"><blockquote \
class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799gmail-m_1444675035890430144quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><br \
clear="none">I'm testing glusterfs before using it in production, it should be \
used to store vm for nodes with libvirtd.<br clear="none"><br clear="none"><div>In \
production I will have 4 nodes connected with a dedicated 20gbit/s \
network.</div><div><br clear="none"></div><div>Which version to use in production on \
a centos 7.x? Should I use Gluster version 6?</div><div><br \
clear="none"></div><div>To make the volume available to libvirtd the best method is \
to use FUSE?</div><div><br clear="none"></div><div>I see that stripped is deprecated. \
Is it reasonable to use the volume with 3 replicas on 4 nodes and sharding enabled? \
<br clear="none"></div><div></div><div>Is there convenience to use sharding volume in \
this context? I think could positive inpact in read performance or rebalance. Is it \
true?<br clear="none"></div><div><br clear="none"></div>In the vm configuration I use \
the virtio disk. How is it better to set the disk cache to get the best performances \
none, default or writeback?<div><br clear="none">Thanks in advance for your patience \
and answers.</div><div><br clear="none"></div><div>Thanks,<br \
clear="none"></div><div><div dir="ltr"><div dir="ltr"><div><br \
clear="none"></div><div><b><font color="#990000"><i><br \
clear="none"></i></font></b></div><div><b><font color="#990000"><i>Cristian Del \
Carlo</i></font></b><br clear="none"></div><div></div></div></div></div></div> \
</blockquote></div></blockquote></div></div><br clear="all"><br clear="none">-- <br \
clear="none"><div class="gmail-m_-1175703377238968109gmail-m_-6718989892495721822gmail-m_-4360627142264084429ydp93be247yiv2327294799gmail_signature" \
dir="ltr"><div dir="ltr"><div><span></span><span></span><br \
clear="none"></div><div><font><b><font color="#990000"><i><br \
clear="none"></i></font></b></font></div><div><font><b><font \
color="#990000"><i>Cristian Del Carlo</i></font></b><br \
clear="none"></font></div><div></div><div><i><font>Target Solutions s.r.l.<br \
clear="none"></font></i></div><div><i><font><br clear="none"></font></i></div>
<div><font><span><b><font color="#ff0000">T</font></b><b> </b>+</span>39 0583 \
1905621</font></div><div><font><font color="#ff0000"><b>F</b></font> <a \
shape="rect">+39 0583 1905675</a></font></div><div> <font><a \
shape="rect"></a></font></div><div><font><font color="#ff0000"><b>@</b></font> \
</font><font><a shape="rect" href="mailto:cristian.delcarlo@targetsolutions.it" \
rel="nofollow" target="_blank">cristian.delcarlo@targetsolutions.it</a></font><br \
clear="none"></div>
<div><br clear="none"><a shape="rect" href="http://www.targetsolutions.it/" \
rel="nofollow" target="_blank"><font>http://www.targetsolutions.it</font></a><br \
clear="none"><font>P.IVA e C.Fiscale: 01815270465 Reg. Imp. di Lucca<br \
clear="none">Capitale Sociale: €11.000,00 iv - REA n 173227<br clear="none"> \
<br clear="none"></font><font size="1">Il testo e gli eventuali documenti trasmessi \
contengono informazioni riservate al destinatario indicato. La seguente e-mail \
e' confidenziale e la sua riservatezza e' tutelata legalmente dal Decreto \
Legislativo 196 del 30/06/2003 (Codice di tutela della privacy). La lettura, copia o \
altro uso non autorizzato o qualsiasi altra azione derivante dalla
conoscenza di queste informazioni sono rigorosamente vietate. Qualora
abbiate ricevuto questo documento per errore siete cortesemente pregati
di darne immediata comunicazione al mittente e di provvedere,
immediatamente, alla sua distruzione.</font></div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br clear="all"><br><br></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" \
target="_blank">Gluster-users@gluster.org</a><br> <a \
href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" \
target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>
</blockquote></div>
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic