[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: Re: Understanding the relationship between block size and RPC / IPC length?
From: Wei-Chiu Chuang <weichiu () cloudera ! com ! INVALID>
Date: 2019-11-08 11:51:02
Message-ID: CADiq6=yxEPXhrTAMK-Me=Bw0tphu=j3fSwkUNuD0daskmHuKmA () mail ! gmail ! com
[Download RAW message or body]
There are more details in this jira:
https://issues.apache.org/jira/browse/HADOOP-16452
Denser DataNodes are common. It is not uncommon to find a DataNode with > 7
> million blocks these days.
> With such a high number of blocks, the block report message can exceed the
> 64mb limit (defined by ipc.maximum.data.length). The block reports are
> rejected, causing missing blocks in HDFS. We had to double this
> configuration value in order to work around the issue.
On Fri, Nov 8, 2019 at 1:48 AM Carey, Paul <Paul.Carey@sc.com.invalid>
wrote:
> Hi
>
>
>
> The NameNode logs in my HDFS instance recently started logging warnings of
> the form `Requested data length 145530837 is longer than maximum configured
> RPC length 144217728`.
>
>
>
> This ultimately manifested itself as the NameNode declaring thousands of
> blocks to be missing and 19 files to be corrupt.
>
>
>
> The situation was resolved by updating `ipc.maximum.data.length` to a
> value greater than the requested data length listed above. This is not a
> satisfying resolution though. I'd like to understand how this issue
> occurred.
>
>
>
> I've run `hdfs fsck -files -blocks -locations` and the largest block is of
> length `1342177728`.
>
>
>
> - Is there some overhead for RPC calls? Could a block of length
> `1342177728` be resulting in the original warning log at the top of this
> post?
>
> - My understanding is that the only way a client writing to HDFS can
> specify a block size is via either `-Ddfs.blocksize` or setting the
> corresponding property on the `Configuration` object when initialising the
> HDFS connection. Is this correct, or are there any other routes to creating
> excessively large blocks?
>
> - Other than overly large blocks, are there any other issues that could
> trigger the warning above?
>
>
>
> Many thanks
>
>
>
> Paul
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at https://www.sc.com/en/our-locations. Please
> refer to https://www.sc.com/en/privacy-policy/ for Standard Chartered
> Bank's Privacy Policy.
>
[Attachment #3 (text/html)]
<div dir="ltr">There are more details in this jira: <a \
href="https://issues.apache.org/jira/browse/HADOOP-16452">https://issues.apache.org/jira/browse/HADOOP-16452</a><div><br></div><div><blockquote \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex" class="gmail_quote">Denser DataNodes are common. \
It is not uncommon to find a DataNode with > 7 million blocks these days.<br>With \
such a high number of blocks, the block report message can exceed the 64mb limit \
(defined by ipc.maximum.data.length). The block reports are rejected, causing missing \
blocks in HDFS. We had to double this configuration value in order to work around the \
issue.</blockquote></div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Fri, Nov 8, 2019 at 1:48 AM Carey, Paul \
<Paul.Carey@sc.com.invalid> wrote:<br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div class="gmail-m_6898225189296807298WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">The NameNode logs in my HDFS instance \
recently started logging warnings of the form `Requested data length 145530837 is \
longer than maximum configured RPC length 144217728`.<u></u><u></u></span></p> <p \
class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <p \
class="MsoNormal"><span lang="EN-US">This ultimately manifested itself as the \
NameNode declaring thousands of blocks to be missing and 19 files to be \
corrupt.<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US"><u></u> \
<u></u></span></p> <p class="MsoNormal"><span lang="EN-US">The situation was resolved \
by updating `ipc.maximum.data.length` to a value greater than the requested data \
length listed above. This is not a satisfying resolution though. I'd like to \
understand how this issue occurred.<u></u><u></u></span></p> <p \
class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <p \
class="MsoNormal"><span lang="EN-US">I've run `hdfs fsck -files -blocks \
-locations` and the largest block is of length `1342177728`.<u></u><u></u></span></p> \
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <p \
class="MsoNormal"><span lang="EN-US">- Is there some overhead for RPC calls? Could a \
block of length `1342177728` be resulting in the original warning log at the top of \
this post?<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US">- My \
understanding is that the only way a client writing to HDFS can specify a block size \
is via either `-Ddfs.blocksize` or setting the corresponding property on the \
`Configuration` object when initialising the HDFS connection. Is this correct, or \
are there any other routes to creating excessively large \
blocks?<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US">- Other than \
overly large blocks, are there any other issues that could trigger the warning \
above?<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US"><u></u> \
<u></u></span></p> <p class="MsoNormal"><span lang="EN-US">Many \
thanks<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US"><u></u> \
<u></u></span></p> <p class="MsoNormal"><span \
lang="EN-US">Paul<u></u><u></u></span></p> </div>
<br clear="both">
This email and any attachments are confidential and may also be privileged. If you \
are not the intended recipient, please delete all copies and notify the sender \
immediately. You may wish to refer to the incorporation details of Standard Chartered \
PLC, Standard Chartered Bank and their subsidiaries at <a \
href="https://www.sc.com/en/our-locations" \
target="_blank">https://www.sc.com/en/our-locations</a>. Please refer to <a \
href="https://www.sc.com/en/privacy-policy/" \
target="_blank">https://www.sc.com/en/privacy-policy/</a> for Standard Chartered \
Bank's Privacy Policy.<br> </div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic