[prev in list] [next in list] [prev in thread] [next in thread]
List: gluster-users
Subject: [Gluster-users] Regarding the issues gluster DHT and Layouts of bricks
From: subrata.ghosh () ericsson ! com (Subrata Ghosh)
Date: 2015-05-26 11:35:17
Message-ID: FAEC14C879684F41B3AA48513E634F9686F0CA () ESESSMB105 ! ericsson ! se
[Download RAW message or body]
Sorry for the wrong incomplete message sent by mistake earlier.
Hi Sussant,
Extremely sorry for the belated reply. Thanks for your input. We will try having \
rebalance then create a set of small files with a random pattern generation and check \
where it falls in by DM_TYPE DHT.
I have one short query:
We were also thinking to use (in case we need ) Translators/cluster/unify along with \
NUFA or ALU scheduler. However we noticed Translators/cluster/unify became \
obsolete/legacy in gluster site , does it mean automatically ALU, NUFA,rr scheduler \
also became obsolete or not supported ? Can you please clarify ?
Is the Translators/cluster/unify is supported in 3.3.2 ( we are currently using) and \
following configuration would work ?
If not can you please suggest you how to use ALU/NUFA scheduler with 3.3.2 if there \
is any specific mechanism?
volume unify
type cluster/unify
subvolumes brick1 brick2 brick3 brick4
option namespace brick-ns # should be a node which is not present in 'subvolumes'
option scheduler NUFA/ALU/rr # NUFA/ALU or Simple round-robin scheduler
end-volume
Thanks again for your time and advice.
Regards,
-----Original Message-----
From: Susant Palai [mailto:spalai at redhat.com]
Sent: Thursday, May 21, 2015 6:17 PM
To: Subrata Ghosh
Cc: gluster-devel at gluster.org; gluster-users at gluster.org; Nobin Mathew; Vijay \
Bellur
Subject: Re: Regarding the issues gluster DHT and Layouts of bricks
Commets inline.
----- Original Message -----
> From: "Subrata Ghosh" <subrata.ghosh at ericsson.com<mailto:subrata.ghosh at \
> ericsson.com>>
> To: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>, \
> gluster-users at gluster.org<mailto:gluster-users at gluster.org>
> Cc: "Nobin Mathew" <nobin.mathew at ericsson.com<mailto:nobin.mathew at \
> ericsson.com>>, "Susant Palai" <spalai at redhat.com<mailto:spalai at redhat.com>>, \
> "Vijay Bellur" <vbellur at redhat.com<mailto:vbellur at redhat.com>>
> Sent: Thursday, 21 May, 2015 4:26:05 PM
> Subject: Regarding the issues gluster DHT and Layouts of bricks
>
>
> Hi All,
>
> Could you please guide us to solve the following DHT and brick layout
> problem we are dealing with ? Questions are marked bold.
>
> Problem statement :
>
>
> 1. We have a requirement to achieve maximum write and read performance
> and we have to meet some committed performance metrics.
>
> Our goal is to place each file into different bricks to get
> optimal performance and also observer the nature of the
> throughput , hence need to have a mechanism to generate
> different hash using gluster glusterfs.gf_dm_hashfn,
> (assuming number of files are : N , Number of Bricks :N) to place
> spate bricks.
>
>
> - How to make sure each file has different hash and falls to
> different bricks ?
>
>
>
> - Other way to put the question if I know the range of the brick
> layout or more precisely if I know the hex value of the desired hash
> ( so that it will be placed desired brick) that we need to generate
> from Davis-Meyer algorithm used in gluster, Can we create a file name
> such that, that also solve our problem to some extent?
>
>
> 2. We tried to experiment to see how a file in gluster is decided to be
> placed in a particular brick following gluster glusterfs.gf_dm_hashfn
> and took some idea from
> some articles like
> http://gluster.readthedocs.org/en/latest/Features/dht/ ,
> https://joejulian.name/blog/dht-misses-are-expensive/ page which
> describes layout for that brick and calculate a hash for the file.
>
>
> To minimize collisions or generating different hash in such way to
> place each file in different bricks ( file 1 => brick A, file 2 =>
> Brick B, file 3=> Brick C, file 4 => brick D)
>
> We use kind of similar script to get the hash value for
> a file
>
> def gf_dm_hashfn(filename):
> return ctypes.c_uint32(glusterfs.gf_dm_hashfn(
> filename,
> len(filendame)))
>
> if __name__ == "__main__":
> print hex(gf_dm_hashfn(sys.argv[1]).value)
>
> We can then calculate the hash for a filename:
> # python gf_dm_hash.py file1
> 0x99d1b6fL
>
>
> Extended attribute is fetch to check the range and try to match the
> above generated hash value.
>
> getfattr -n trusted.glusterfs.dht -e hex file1
>
>
> However we are not able to exactly follow till this point , how the
> hash value matched to one of the layout assignments, to yield what we
> call a hashed location.
>
>
> - My question is if I know the range of brick lay out ( say
> 0xc0000000 to 0xffffffff, is range select a hash 0xc0070000 ) where
> to be placed the next file can we generate the name ( kind of reverse
> of gluster
> glusterfs.gf_dm_hashfn) ?
I am not aware of any such mechanism. You will have to generate file names manually \
and run them through your script to check whether it falls in the brick range.
>
> PS : Susant : Can you throw some light or suggest a method we are
> trying to solve.
>
> Thanks for your time.
>
>
> Best Regards,
> Subrata Ghosh
>
>
>
>
>
>
>
Regards,
Susant
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150526/ba6c8ae6/attachment.html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic