[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Fwd: FileUtil.fullyDelete does ?
From:       Divya Gehlot <divya.htconex () gmail ! com>
Date:       2016-07-26 11:09:15
Message-ID: CAHfzKEqQfeGmgLCuOk23C-29Czs=w5CZyz05aNv2xyJajij20w () mail ! gmail ! com
[Download RAW message or body]

Resending to right list
---------- Forwarded message ----------
From: "Divya Gehlot" <divya.htconex@gmail.com>
Date: Jul 26, 2016 6:51 PM
Subject: FileUtil.fullyDelete does ?
To: "user @spark" <user@spark.apache.org>
Cc:

Hi,
When I am doing the using theFileUtil.copymerge function

val file = "/tmp/primaryTypes.csv"

FileUtil.fullyDelete(new File(file))

 val destinationFile= "/tmp/singlePrimaryTypes.csv"

FileUtil.fullyDelete(new File(destinationFile))

 val counts = partitions.

reduceByKey {case (x,y) => x + y}.

sortBy {case (key, value) => -value}.

map { case (key, value) => Array(key, value).mkString(",") }

 counts.saveAsTextFile(file)

 merge(file, destinationFile)


I am wondering here what does  FileUtil.fullyDelete(new
File(destinationFile)) do ?

  does it delete the merged file ....If yes,then how will we access
the merged file ..?


Confused here ...



Thanks,

Divya

[Attachment #3 (text/html)]

<p dir="ltr">Resending to right list </p>
<div class="gmail_quote">---------- Forwarded message ----------<br>From: &quot;Divya \
Gehlot&quot; &lt;<a href="mailto:divya.htconex@gmail.com">divya.htconex@gmail.com</a>&gt;<br>Date: \
Jul 26, 2016 6:51 PM<br>Subject: FileUtil.fullyDelete does ?<br>To: &quot;user \
@spark&quot; &lt;<a href="mailto:user@spark.apache.org">user@spark.apache.org</a>&gt;<br>Cc: \
<br><br type="attribution"><div dir="ltr">Hi,<div>When I am doing the using \
theFileUtil.copymerge function  </div><div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>val</span> <span>file</span> <span>=</span> <span \
style="color:rgb(170,17,17)">&quot;/tmp/primaryTypes.csv&quot;</span></span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>FileUtil</span>.<span>fullyDelete</span>(<span \
style="color:rgb(119,0,136)">new</span> \
<span>File</span>(<span>file</span>))</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"> </span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>val</span> \
<span>destinationFile</span><span>=</span> <span \
style="color:rgb(170,17,17)">&quot;/tmp/singlePrimaryTypes.csv&quot;</span></span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>FileUtil</span>.<span>fullyDelete</span>(<span \
style="color:rgb(119,0,136)">new</span> \
<span>File</span>(<span>destinationFile</span>))</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"> </span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>val</span> <span>counts</span> <span>=</span> \
<span>partitions</span>.</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>reduceByKey</span> {<span \
style="color:rgb(119,0,136)">case</span> (<span>x</span>,<span>y</span>) \
<span>=&gt;</span> <span>x</span> <span>+</span> \
<span>y</span>}.</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>sortBy</span> {<span \
style="color:rgb(119,0,136)">case</span> (<span>key</span>, <span>value</span>) \
<span>=&gt;</span> <span>-</span><span>value</span>}.</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>map</span> { <span \
style="color:rgb(119,0,136)">case</span> (<span>key</span>, <span>value</span>) \
<span>=&gt;</span> <span>Array</span>(<span>key</span>, \
<span>value</span>).<span>mkString</span>(<span \
style="color:rgb(170,17,17)">&quot;,&quot;</span>) }</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"> </span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"><span>counts</span>.<span>saveAsTextFile</span>(<span>file</span>)</span></pre></div><div \
style="font-family:monospace;font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="width:29px;border:none!important;background:none!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><span \
style="padding-right:29px"> </span></pre></div><div \
style="font-size:13px;color:rgb(0,0,0);line-height:18.85px;background-color:rgb(252,252,252)"><div \
style="font-family:monospace;width:29px;border:none!important;background-image:none!im \
portant;background-color:initial!important;background-position:initial!important;background-repeat:initial!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="font-family:monospace;overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><span \
style="padding-right:29px"><span>merge</span>(<span>file</span>, \
<span>destinationFile</span>)</span></pre><pre \
style="font-family:monospace;overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent"><br></span></pre><pre \
style="font-family:monospace;overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">I \
am wondering here what does  FileUtil</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">.</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">fullyDelete</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">(</span><span \
style="font-family:arial,helvetica,sans-serif;line-height:inherit;color:rgb(119,0,136);background-color:transparent">new</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent"> \
</span><span style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">File</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">(</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">destinationFile</span><span \
style="font-family:arial,helvetica,sans-serif;color:inherit;line-height:inherit;background-color:transparent">)) \
do ?</span><br></pre><div style="line-height:18.85px"><div \
style="width:29px;border:none!important;background-image:none!important;background-col \
or:initial!important;background-position:initial!important;background-repeat:initial!important"><div \
style="padding:0px 3px 0px \
5px;min-width:20px;text-align:right;color:rgb(153,153,153);white-space:nowrap;width:20px"></div></div><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><span \
style="padding-right:29px"><font face="arial, helvetica, sans-serif">  \
</font></span></pre><pre style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><span \
style="padding-right:29px"><font face="arial, helvetica, sans-serif">does it delete \
the merged file ....If yes,then how will we access the merged file \
..?</font></span></pre><pre style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><span \
style="padding-right:29px"><font face="arial, helvetica, \
sans-serif"><br></font></span></pre><pre style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><font \
face="arial, helvetica, sans-serif">Confused here ...</font></pre><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><font \
face="arial, helvetica, sans-serif"><br></font></pre><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><font \
face="arial, helvetica, sans-serif"><br></font></pre><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><font \
face="arial, helvetica, sans-serif">Thanks,</font></pre><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background-image:initial;background-color:transparent;background-position:initial;background-repeat:initial"><font \
face="arial, helvetica, sans-serif">Divya </font></pre><pre \
style="overflow:visible;padding:0px \
4px;margin-top:0px;margin-bottom:0px;line-height:inherit;word-wrap:normal;color:inherit;border:0px \
solid rgb(204,204,204);border-radius:0px;background:transparent"><br></pre></div></div></div></div>
 </div>



[prev in list] [next in list] [prev in thread] [next in thread]

Configure | About | News | Add a list | Sponsored by KoreLogic