[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: Re: Can block size for namenode be different from wdatanode block size?
From: Harsh J <harsh () cloudera ! com>
Date: 2015-03-25 21:31:38
Message-ID: CAOcnVr2NiPG+kcxqsHp=LgrvW9Wtz8-wNd0GTNRxpVq=yZhDqg () mail ! gmail ! com
[Download RAW message or body]
> 2. The block size is only relevant to DataNodes (DN). NameNode (NN)
does not use this parameter
Actually, as a configuration, its only relevant to the client. See also
http://www.quora.com/How-do-I-check-HDFS-blocksize-default-custom
Other points sound about right, except the ability to do (7) can only now
be done if you have legacy mode of fsimage writes enabled. The new OIV tool
in recent releases only serves a REST based Web Server to query the file
data upon.
On Thu, Mar 26, 2015 at 1:47 AM, Mich Talebzadeh <mich@peridale.co.uk>
wrote:
> Thank you all for your contribution.
>
>
>
> I have summarised the findings as below
>
>
>
> 1. The Hadoop block size is a configurable parameter dfs.block.size
> in bytes . By default this is set to 134217728 bytes or 128MB
>
> 2. The block size is only relevant to DataNodes (DN). NameNode (NN)
> does not use this parameter
>
> 3. NN behaves like an in-memory database IMDB and uses a disk file
> system called the FsImage to load the metadata as startup. This is the only
> place that I see value for Solid State Disk to make this initial load faster
>
> 4. For the remaining period until HDFS shutdown or otherwise NN will
> use the in memory cache to access metadata
>
> 5. With regard to sizing of NN to store metadata, one can use the
> following rules of thumb (heuristics):
>
> a. NN consumes roughly 1GB for every 1 million blokes (source Hadoop
> Operations, Eric Sammer, ISBN: 978-1-499-3205-7). So if you have 128MB
> block size, you can store 128 * 1E6 / (3 *1024) = 41,666GB of data for
> every 1GB. Number 3 comes from the fact that the block is replicated three
> times. In other words just under 42TB of data. So if you have 10GB of
> namenode cache, you can have up to 420TB of data on your datanodes
>
> 6. You can take FsImage file from Hadoop and convert it into a text
> file as follows:
>
>
>
> *hdfs dfsadmin -fetchImage nnimage*
>
>
>
> 15/03/25 20:17:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 15/03/25 20:17:41 INFO namenode.TransferFsImage: Opening connection to
> http://rhes564:50070/imagetransfer?getimage=1&txid=latest
>
> 15/03/25 20:17:41 INFO namenode.TransferFsImage: Image Transfer timeout
> configured to 60000 milliseconds
>
> 15/03/25 20:17:41 WARN namenode.TransferFsImage: Overwriting existing file
> nnimage with file downloaded from
> http://rhes564:50070/imagetransfer?getimage=1&txid=latest
>
> 15/03/25 20:17:41 INFO namenode.TransferFsImage: Transfer took 0.03s at
> 1393.94 KB/s
>
>
>
> 7. That create an image file in the current directory that can be
> converted to text file
>
> *hdfs oiv -i nnimage -o nnimage.txt*
>
>
>
> 15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: Loading 2 strings
>
> 15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: Loading 543
> inodes.
>
> 15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: Loading inode
> references
>
> 15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: Loaded 0 inode
> references
>
> 15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: Loading inode
> directory section
>
> 15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: Loaded 198
> directories
>
> 15/03/25 20:20:07 INFO offlineImageViewer.WebImageViewer: WebImageViewer
> started. Listening on /127.0.0.1:5978. Press Ctrl+C to stop the viewer.
>
>
>
> Let me know if I missed anything or got it wrong.
>
>
>
> HTH
>
>
>
> Mich Talebzadeh
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Publications due shortly:*
>
> *Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and
> Coherence Cache*
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Ltd, its
> subsidiaries or their employees, unless expressly so stated. It is the
> responsibility of the recipient to ensure that this email is virus free,
> therefore neither Peridale Ltd, its subsidiaries nor their employees accept
> any responsibility.
>
>
>
--
Harsh J
[Attachment #3 (text/html)]
<div dir="ltr">> <span \
style="font-size:10pt;font-family:Arial,sans-serif">2.<span \
style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'"> \
</span></span><u style="font-size:12.8000001907349px"></u><span \
style="font-size:10pt;font-family:Arial,sans-serif">The block size is only relevant \
to DataNodes (DN). NameNode (NN) does not use this parameter</span><div><span \
style="font-size:10pt;font-family:Arial,sans-serif"><br></span></div><div><span \
style="font-size:10pt;font-family:Arial,sans-serif">Actually, as a configuration, its \
only relevant to the client. See also </span><font face="Arial, sans-serif"><span \
style="font-size:13.3333330154419px"><a \
href="http://www.quora.com/How-do-I-check-HDFS-blocksize-default-custom">http://www.qu \
ora.com/How-do-I-check-HDFS-blocksize-default-custom</a></span></font></div><div><font \
face="Arial, sans-serif"><span \
style="font-size:13.3333330154419px"><br></span></font></div><div><font face="Arial, \
sans-serif"><span style="font-size:13.3333330154419px">Other points sound about \
right, except the ability to do (7) can only now be done if you have legacy mode of \
fsimage writes enabled. The new OIV tool in recent releases only serves a REST based \
Web Server to query the file data upon.</span></font></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 26, 2015 at 1:47 AM, \
Mich Talebzadeh <span dir="ltr"><<a href="mailto:mich@peridale.co.uk" \
target="_blank">mich@peridale.co.uk</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div lang="EN-GB" link="blue" vlink="purple"><div><p \
class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Thank \
you all for your contribution.<u></u><u></u></span></p><p class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">I have \
summarised the findings as below<u></u><u></u></span></p><p class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>1.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">The \
Hadoop block size is a configurable parameter dfs.block.size in bytes . By default \
this is set to 134217728 bytes or 128MB<u></u><u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>2.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">The \
block size is only relevant to DataNodes (DN). NameNode (NN) does not use this \
parameter<u></u><u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>3.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">NN \
behaves like an in-memory database IMDB and uses a disk file system called the \
FsImage to load the metadata as startup. This is the only place that I see value for \
Solid State Disk to make this initial load \
faster<u></u><u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>4.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">For the \
remaining period until HDFS shutdown or otherwise NN will use the in memory cache to \
access metadata<u></u><u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>5.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">With \
regard to sizing of NN to store metadata, one can use the following rules of thumb \
(heuristics):<u></u><u></u></span></p><p style="margin-left:72.0pt"><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>a.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">NN \
consumes roughly 1GB for every 1 million blokes (source Hadoop Operations, Eric \
Sammer, ISBN: 978-1-499-3205-7). So if you have 128MB block size, you can store 128 \
* 1E6 / (3 *1024) = 41,666GB of data for every 1GB. Number 3 comes from the fact that \
the block is replicated three times. In other words just under 42TB of data. So if \
you have 10GB of namenode cache, you can have up to 420TB of data on your \
datanodes<u></u><u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>6.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">You can \
take FsImage file from Hadoop and convert it into a text file as \
follows:<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p><p class="MsoNormal" style="text-indent:36.0pt"><b><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue">hdfs \
dfsadmin -fetchImage nnimage<u></u><u></u></span></b></p><p class="MsoNormal" \
style="text-indent:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue"><u></u> \
<u></u></span></p><p class="MsoNormal" style="text-indent:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue">15/03/25 \
20:17:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your \
platform... using builtin-java classes where applicable<u></u><u></u></span></p><p \
class="MsoNormal" style="text-indent:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue">15/03/25 \
20:17:41 INFO namenode.TransferFsImage: Opening connection to <a \
href="http://rhes564:50070/imagetransfer?getimage=1&txid=latest" \
target="_blank">http://rhes564:50070/imagetransfer?getimage=1&txid=latest</a><u></u><u></u></span></p><p \
class="MsoNormal" style="text-indent:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue">15/03/25 \
20:17:41 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 \
milliseconds<u></u><u></u></span></p><p class="MsoNormal" \
style="text-indent:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue">15/03/25 \
20:17:41 WARN namenode.TransferFsImage: Overwriting existing file nnimage with file \
downloaded from <a href="http://rhes564:50070/imagetransfer?getimage=1&txid=latest" \
target="_blank">http://rhes564:50070/imagetransfer?getimage=1&txid=latest</a><u></u><u></u></span></p><p \
class="MsoNormal" style="text-indent:36.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue">15/03/25 \
20:17:41 INFO namenode.TransferFsImage: Transfer took 0.03s at 1393.94 \
KB/s<u></u><u></u></span></p><p style="margin-left:18.0pt"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p><p><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span>7.<span \
style="font:7.0pt "Times New Roman""> \
</span></span></span><u></u><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">That \
create an image file in the current directory that can be converted to text \
file<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:36.0pt"><b><span \
style="font-size:10.0pt;font-family:"Courier New";color:blue">hdfs oiv -i \
nnimage -o nnimage.txt<u></u><u></u></span></b></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue"><u></u> <u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: \
Loading 2 strings<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: \
Loading 543 inodes.<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: \
Loading inode references<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: \
Loaded 0 inode references<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: \
Loading inode directory section<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.FSImageHandler: \
Loaded 198 directories<u></u><u></u></span></p><p class="MsoNormal" \
style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier \
New";color:blue">15/03/25 20:20:07 INFO offlineImageViewer.WebImageViewer: \
WebImageViewer started. Listening on /<a href="http://127.0.0.1:5978" \
target="_blank">127.0.0.1:5978</a>. Press Ctrl+C to stop the \
viewer.<u></u><u></u></span></p><p style="margin-left:72.0pt"><span \
style="font-size:10.0pt;font-family:"Courier New";color:blue"><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Let me \
know if I missed anything or got it wrong.<u></u><u></u></span></p><p \
class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif"">HTH<u></u><u></u></span></p><p \
class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Arial","sans-serif";color:black">Mich \
Talebzadeh<u></u><u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Arial","sans-serif";color:black"><u></u> \
<u></u></span></p><p class="MsoNormal" style="text-align:justify"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black"><a \
href="http://talebzadehmich.wordpress.com" \
target="_blank">http://talebzadehmich.wordpress.com</a><u></u><u></u></span></p><p \
class="MsoNormal" style="text-align:justify"><u><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black;letter-spacing:-.15pt"><u></u><span \
style="text-decoration:none"> </span><u></u></span></u></p><p class="MsoNormal" \
style="text-align:justify"><u><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black;letter-spacing:-.15pt">Publications \
due shortly:<u></u><u></u></span></u></p><p class="MsoNormal" \
style="text-align:justify"><b><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Creating \
in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence \
Cache<u></u><u></u></span></b></p><p class="MsoNormal" \
style="text-align:justify"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black"><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">NOTE: \
The information in this email is proprietary and confidential. This message is for \
the designated recipient only, if you are not the intended recipient, you should \
destroy it immediately. Any information in this message shall not be understood as \
given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless \
expressly so stated. It is the responsibility of the recipient to ensure that this \
email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their \
employees accept any responsibility.</span><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"><u></u><u></u></span></p><p \
class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Arial","sans-serif""><u></u> \
<u></u></span></p></div></div></blockquote></div><br><br \
clear="all"><div><br></div>-- <br><div class="gmail_signature">Harsh J</div> </div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic