[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: Re: Unable to append to a file in HDFS
From: Ravi Prakash <ravihadoop () gmail ! com>
Date: 2017-10-31 22:32:36
Message-ID: CAMs9kViE+0JNrT_s4TyfQ75X+81KjrPpPwoNFt+G=dq8ax7DwQ () mail ! gmail ! com
[Download RAW message or body]
HI Tarik!
I'm glad you were able to diagnose your issue. Thanks for sharing with the
user list. I suspect your writer may have set minimum replication to 3, and
since you have only 2 datanodes, the Namenode will not allow you to
successfully close the file. You could add another node or reduce the
minimum replication.
HTH,
Ravi
On Mon, Oct 30, 2017 at 3:13 PM, Tarik Courdy <tarik.courdy@gmail.com>
wrote:
> Hello Ravi -
>
> I have pin pointed my issue a little more. When I create a file with a
> dfs.replication factor of 3 I can never append. However, if I create a
> file with a dfs.replication factor of 1 then I can append to the file all
> day long.
>
> Thanks again for your help regarding this.
>
> -Tarik
>
> On Mon, Oct 30, 2017 at 2:46 PM, Tarik Courdy <tarik.courdy@gmail.com>
> wrote:
>
> > Hello Ravi -
> >
> > I greped the directory that has my logs and couldn't find any instance of
> > "NameNode.complete".
> >
> > I just created a new file in hdfs using hdfs -touchz and it is allowing
> > me to append to it with no problem.
> >
> > Not sure who is holding the eternal lease on my first file.
> >
> > Thanks again for your time.
> >
> > -Tarik
> >
> > On Mon, Oct 30, 2017 at 2:19 PM, Ravi Prakash <ravihadoop@gmail.com>
> > wrote:
> >
> > > Hi Tarik!
> > >
> > > You're welcome! If you look at the namenode logs, do you see a "DIR*
> > > NameNode.complete: " message ? It should have been written when the first
> > > client called close().
> > >
> > > Cheers
> > > Ravi
> > >
> > > On Mon, Oct 30, 2017 at 1:13 PM, Tarik Courdy <tarik.courdy@gmail.com>
> > > wrote:
> > >
> > > > Hello Ravi -
> > > >
> > > > Thank you for your response. I have read about the soft and hard lease
> > > > limits, however no matter how long I wait I am never able to write again to
> > > > the file that I first created and wrote to the first time.
> > > >
> > > > Thanks again.
> > > >
> > > > -Tarik
> > > >
> > > > On Mon, Oct 30, 2017 at 2:08 PM, Ravi Prakash <ravihadoop@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Tarik!
> > > > >
> > > > > The lease is owned by a client. If you launch 2 client programs, they
> > > > > will be viewed as separate (even though the user is same). Are you sure you
> > > > > closed the file when you first wrote it? Did the client program which wrote
> > > > > the file, exit cleanly? In any case, after the namenode lease hard
> > > > > timeout
> > > > > <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java#L82>,
> > > > > the lease will be recovered, and you ought to be able to append to it. Is
> > > > > that not what you are seeing?
> > > > >
> > > > > HTH
> > > > > Ravi
> > > > >
> > > > > On Mon, Oct 30, 2017 at 11:04 AM, Tarik Courdy <tarik.courdy@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Good morning -
> > > > > >
> > > > > > I have a file in hdfs that I can write to once but when I try to
> > > > > > append to it I receive an error stating that someone else owns the file
> > > > > > lease.
> > > > > >
> > > > > > I am the only one trying to append to this file. I have also made
> > > > > > sure that dfs.support.append has been set to true. Additionally, I have
> > > > > > also tried setting the the dfs.replication to 1 since I read this had
> > > > > > helped someone else with this issue.
> > > > > >
> > > > > > However, neither of these have allowed me to append to the file.
> > > > > >
> > > > > > My HDFS setup consists of a name node, a secondary name node, and 2
> > > > > > data nodes.
> > > > > >
> > > > > > Any suggestions that you might be able to provide would be greatly
> > > > > > appreciated.
> > > > > >
> > > > > > Thank you for your time.
> > > > > >
> > > > > > -Tarik
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
[Attachment #3 (text/html)]
<div dir="ltr"><div><div><div>HI Tarik!<br><br></div>I'm glad you were able to \
diagnose your issue. Thanks for sharing with the user list. I suspect your writer may \
have set minimum replication to 3, and since you have only 2 datanodes, the Namenode \
will not allow you to successfully close the file. You could add another node or \
reduce the minimum replication. <br><br></div>HTH,<br></div>Ravi<br></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 3:13 PM, \
Tarik Courdy <span dir="ltr"><<a href="mailto:tarik.courdy@gmail.com" \
target="_blank">tarik.courdy@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-family:monospace,monospace">Hello Ravi - </div><div \
class="gmail_default" style="font-family:monospace,monospace"><br></div><div \
class="gmail_default" style="font-family:monospace,monospace">I have pin pointed my \
issue a little more. When I create a file with a dfs.replication factor of 3 I can \
never append. However, if I create a file with a dfs.replication factor of 1 then I \
can append to the file all day long.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">Thanks again for your help regarding \
this.</div><span class="HOEnZb"><font color="#888888"><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">-Tarik</div></font></span></div><div \
class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div \
class="gmail_quote">On Mon, Oct 30, 2017 at 2:46 PM, Tarik Courdy <span \
dir="ltr"><<a href="mailto:tarik.courdy@gmail.com" \
target="_blank">tarik.courdy@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-family:monospace,monospace">Hello Ravi - </div><div \
class="gmail_default" style="font-family:monospace,monospace"><br></div><div \
class="gmail_default" style="font-family:monospace,monospace">I greped the directory \
that has my logs and couldn't find any instance of \
"NameNode.complete".</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">I just created a new file in hdfs using hdfs \
-touchz and it is allowing me to append to it with no problem.</div><div \
class="gmail_default" style="font-family:monospace,monospace"><br></div><div \
class="gmail_default" style="font-family:monospace,monospace">Not sure who is holding \
the eternal lease on my first file.</div><span><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">Thanks again for your time.</div><div \
class="gmail_default" style="font-family:monospace,monospace"><br></div></span><span \
class="m_192613600774258508HOEnZb"><font color="#888888"><div class="gmail_default" \
style="font-family:monospace,monospace">-Tarik</div></font></span></div><div \
class="m_192613600774258508HOEnZb"><div class="m_192613600774258508h5"><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 2:19 PM, \
Ravi Prakash <span dir="ltr"><<a href="mailto:ravihadoop@gmail.com" \
target="_blank">ravihadoop@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi \
Tarik!<br><br></div>You're welcome! If you look at the namenode logs, do you see \
a "DIR* NameNode.complete: " message ? It should have been written when \
the first client called close().<br><br></div>Cheers<span \
class="m_192613600774258508m_3750156454032406755HOEnZb"><font \
color="#888888"><br></font></span></div><span \
class="m_192613600774258508m_3750156454032406755HOEnZb"><font \
color="#888888">Ravi<br></font></span></div><div \
class="m_192613600774258508m_3750156454032406755HOEnZb"><div \
class="m_192613600774258508m_3750156454032406755h5"><div class="gmail_extra"><br><div \
class="gmail_quote">On Mon, Oct 30, 2017 at 1:13 PM, Tarik Courdy <span \
dir="ltr"><<a href="mailto:tarik.courdy@gmail.com" \
target="_blank">tarik.courdy@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-family:monospace,monospace">Hello Ravi - </div><div \
class="gmail_default" style="font-family:monospace,monospace"><br></div><div \
class="gmail_default" style="font-family:monospace,monospace">Thank you for your \
response. I have read about the soft and hard lease limits, however no matter how \
long I wait I am never able to write again to the file that I first created and wrote \
to the first time.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">Thanks again.</div><span \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387HOEnZb"><font \
color="#888888"><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">-Tarik</div></font></span></div><div \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387HOEnZb"><div \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387h5"><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 2:08 PM, \
Ravi Prakash <span dir="ltr"><<a href="mailto:ravihadoop@gmail.com" \
target="_blank">ravihadoop@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Tarik!<br><br></div>The \
lease is owned by a client. If you launch 2 client programs, they will be viewed as \
separate (even though the user is same). Are you sure you closed the file when you \
first wrote it? Did the client program which wrote the file, exit cleanly? In any \
case, after the <a href="https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-proje \
ct/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java#L82" \
target="_blank">namenode lease hard timeout</a>, the lease will be recovered, and you \
ought to be able to append to it. Is that not what you are \
seeing?<br><br></div>HTH<span \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387m_7769358518386498503HOEnZb"><font \
color="#888888"><br></font></span></div><span \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387m_7769358518386498503HOEnZb"><font \
color="#888888">Ravi<br></font></span></div><div \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387m_7769358518386498503HOEnZb"><div \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387m_7769358518386498503h5"><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 11:04 AM, \
Tarik Courdy <span dir="ltr"><<a href="mailto:tarik.courdy@gmail.com" \
target="_blank">tarik.courdy@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-family:monospace,monospace">Good morning - </div><div \
class="gmail_default" style="font-family:monospace,monospace"><br></div><div \
class="gmail_default" style="font-family:monospace,monospace">I have a file in hdfs \
that I can write to once but when I try to append to it I receive an error stating \
that someone else owns the file lease.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">I am the only one trying to append to this \
file. I have also made sure that dfs.support.append has been set to true. \
Additionally, I have also tried setting the the dfs.replication to 1 since I read \
this had helped someone else with this issue.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">However, neither of these have allowed me to \
append to the file.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">My HDFS setup consists of a name node, a \
secondary name node, and 2 data nodes.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">Any suggestions that you might be able to \
provide would be greatly appreciated.</div><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">Thank you for your time.</div><span \
class="m_192613600774258508m_3750156454032406755m_-5081881203755139387m_7769358518386498503m_2355588303270370718HOEnZb"><font \
color="#888888"><div class="gmail_default" \
style="font-family:monospace,monospace"><br></div><div class="gmail_default" \
style="font-family:monospace,monospace">-Tarik</div></font></span></div> \
</blockquote></div><br></div> </div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic