[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: RE: Wrong Root Directory in HDFS API
From: Takanobu Asanuma <tasanuma () yahoo-corp ! jp>
Date: 2018-05-29 9:14:44
Message-ID: OSAPR01MB22763C18A12409847C54C1198D6D0 () OSAPR01MB2276 ! jpnprd01 ! prod ! outlook ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hi Soheil,
That is a correct behavior. The full qualified path of you home directory in HDFS is \
"{fs.defaultFS} + /user/myUserName" and getHomeDirectory complements it.
You can access paths by a few ways.
======
Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://nn-hostname");
FileSystem fs = FileSystem.get(conf);
// your home directory
fs.getFileStatus(new Path(".")) // relative path
fs.getFileStatus(new Path("/user/myUserName")) // absolute path
fs.getFileStatus(new Path("hdfs://nn-hostname/user/myUserName")) // full path
// root directory
fs.getFileStatus(new Path("/")) // absolute path
fs.getFileStatus(new Path("hdfs://nn-hostname/")) // full path
======
Thanks,
- Takanobu
From: Soheil Pourbafrani [mailto:soheil.ir08@gmail.com]
Sent: Monday, May 28, 2018 7:55 PM
To: user@hadoop.apache.org
Subject: Wrong Root Directory in HDFS API
Hi,
I use HDFS java API to do some process. I pass HDFS configuration file (core-site.xml \
and hdfs-site.xml) as Configuration. My Hadoop property
fs.defaultFS
has the value
hdfs://ha-cluster
but when I get HDFS home directory like the following:
FileSystem.get(conf).getHomeDirectory();
It has the value
hdfs://ha-cluster/user/myUserName
!!!
What is the reason and how can I set correct home directory?
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"ＭＳ ゴシック";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"ＭＳ Ｐゴシック";
panose-1:2 11 6 0 7 2 5 8 2 4;}
@font-face
{font-family:"Noto Sans Mono CJK KR Bold";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@ＭＳ ゴシック";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"\@ＭＳ Ｐゴシック";
panose-1:2 11 6 0 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0mm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"ＭＳ Ｐゴシック";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML 書式付き \(文字\)";
margin:0mm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"ＭＳ ゴシック";}
span.HTML
{mso-style-name:"HTML 書式付き \(文字\)";
mso-style-priority:99;
mso-style-link:"HTML 書式付き";
font-family:"Courier New";}
span.19
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Arial",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:99.25pt 30.0mm 30.0mm 30.0mm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026">
<v:textbox inset="5.85pt,.7pt,5.85pt,.7pt" />
</o:shapedefaults></xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="JA" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Hi \
Soheil,<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">That \
is a correct behavior. The full qualified path of you home directory in HDFS is \
"{fs.defaultFS} + /user/myUserName" and getHomeDirectory complements \
it.<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">You \
can access paths by a few ways.<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">======<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Configuration \
conf = new Configuration();<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1 \
F497D">conf.set("fs.defaultFS","hdfs://nn-hostname");<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">FileSystem \
fs = FileSystem.get(conf);<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">// \
your home directory<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">fs.getFileStatus(new \
Path(".")) // relative path<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">fs.getFileStatus(new \
Path("/user/myUserName")) // absolute path<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">fs.getFileStatus(new \
Path("hdfs://nn-hostname/user/myUserName")) // full \
path<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">// \
root directory<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">fs.getFileStatus(new \
Path("/")) // absolute path<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">fs.getFileStatus(new \
Path("hdfs://nn-hostname/")) // full path<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">======<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">- \
Takanobu<o:p></o:p></span></p> <p class="MsoNormal"><a name="_MailEndCompose"><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0mm 0mm 0mm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0mm 0mm 0mm">
<p class="MsoNormal"><b><span lang="EN-US" \
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span \
lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> \
Soheil Pourbafrani [mailto:soheil.ir08@gmail.com] <br>
<b>Sent:</b> Monday, May 28, 2018 7:55 PM<br>
<b>To:</b> user@hadoop.apache.org<br>
<b>Subject:</b> Wrong Root Directory in HDFS API<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi, <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I use HDFS java API to do some process. I \
pass HDFS configuration file (core-site.xml and hdfs-site.xml) as Configuration. My \
Hadoop property <o:p></o:p></span></p> <pre style="background:#2B2B2B"><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Noto Sans Mono CJK KR \
Bold",serif;color:#A9B7C6">fs.defaultFS<o:p></o:p></span></pre> </div>
<div>
<p class="MsoNormal"><span lang="EN-US">has the value <o:p></o:p></span></p>
</div>
<div>
<pre style="background:#2B2B2B"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Noto Sans Mono CJK KR \
Bold",serif;color:#A9B7C6">hdfs://ha-cluster<o:p></o:p></span></pre> <p \
class="MsoNormal"><span lang="EN-US">but when I get HDFS home directory like the \
following:<o:p></o:p></span></p> </div>
<div>
<pre style="background:#2B2B2B"><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Noto Sans Mono CJK KR \
Bold",serif;color:#A9B7C6">FileSystem.<i>get</i>(conf).getHomeDirectory()</span><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Noto Sans Mono CJK KR \
Bold",serif;color:#CC7832">;</span><span lang="EN-US" \
style="font-size:10.0pt;font-family:"Noto Sans Mono CJK KR \
Bold",serif;color:#A9B7C6"><o:p></o:p></span></pre> <p class="MsoNormal"><span \
lang="EN-US">It has the value <o:p></o:p></span></p> </div>
<div>
<pre style="background:#2B2B2B;font-variant-ligatures:normal;font-variant-caps:normal; \
text-align:start;text-decoration-style:initial;text-decoration-color:initial;word-spacing:0px"><span \
lang="EN-US" style="font-size:10.0pt;font-family:"Noto Sans Mono CJK KR \
Bold",serif;color:#A9B7C6">hdfs://ha-cluster/user/myUserName<o:p></o:p></span></pre>
<p class="MsoNormal"><span lang="EN-US">!!!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">What is the reason and how can I set correct \
home directory?<o:p></o:p></span></p> </div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic