[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: Mesos slave not starting up
From: Vinod Kone <vinodkone () gmail ! com>
Date: 2013-08-12 17:05:09
Message-ID: CAAkWvAzzytkU6hGV74CY8MCon8C5ORCzXbWzPcrffz1ExR2hUg () mail ! gmail ! com
[Download RAW message or body]
We have pulled hadoop mesos framework out of mesos repo, to make it easy to
for people to contribute and also to streamline its usage. This would be
the only supported hadoop on mesos port going forward. Please give it a try
and let us know how it works for you.
On Sun, Aug 11, 2013 at 11:24 PM, Johnas, Nalini <njohnas@ebay.com> wrote:
> ** **
>
> Thanks Vinod.****
>
> ** **
>
> Sure will do. May I ask what=92s different with this?****
>
> ** **
>
> -Nalini****
>
> ** **
>
> *From:* vinod@twitter.com [mailto:vinod@twitter.com] *On Behalf Of *Vinod
> Kone
> *Sent:* Sunday, August 11, 2013 8:35 PM
> *To:* user@mesos.apache.org
>
> *Subject:* Re: Mesos slave not starting up****
>
> ** **
>
> Can you try our new instructions at https://github.com/mesos/hadoop ?****
>
> ** **
>
> On Sun, Aug 11, 2013 at 7:19 PM, Johnas, Nalini <njohnas@ebay.com> wrote:=
*
> ***
>
> Hi Vinod,****
>
> ****
>
> I tried everything suggested, still running into the same problem with
> TASK LOST and there is no executor logs created.****
>
> ****
>
> One quick question, Is there any restriction with the Hadoop location ,
> does it need to be under mesos build directory.****
>
> ****
>
> Here=92s where I have these installed.****
>
> ****
>
> Mesos build is under : /home/njohnas_dev/mesos-testing/build****
>
> Hadoop home is under : /home/njohnas_dev/mesos-testing/hadoop****
>
> ****
>
> Also I don=92t mind driving up to a Starbucks closer to you, if you can
> spare like 1/2 hour with your busy schedule to go over my setup and help
> resolve this issue. (or) open to other suggestions as well. Let me know.*=
*
> **
>
> ****
>
> Thanks****
>
> Nalini****
>
> ****
>
> *From:* Johnas, Nalini [mailto:njohnas@ebay.com]
> *Sent:* Tuesday, August 06, 2013 11:59 PM
> *To:* <user@mesos.apache.org>
> *Cc:* user@mesos.apache.org
> *Subject:* Re: Mesos slave not starting up****
>
> ****
>
> Thanks Vinod that's helpful. I suspect it could be the hadoop path. Let
> me give this a try.****
>
> ****
>
> Nalini
>
> Sent from my iPad****
>
>
> On Aug 6, 2013, at 11:52 PM, "Vinod Kone" <vinodkone@gmail.com> wrote:***=
*
>
> An executor terminated as soon as it's launched is indicative of slave
> being unable to fetch/launch the executor.****
>
> ****
>
> In the case of hadoop framework, If your executor sandbox doesn't have a
> hadoop.tar.gz or hadoop directory, that means the slave is unable to fetc=
h
> the executor. It likely means the hdfs url for the executor specified in
> mapred-site.xml is wrong or inaccessible to the slave. ****
>
> ****
>
> Also ensure that 'hadoop' command is in the PATH of the slave (or
> specified via --hadoop_home slave flag), because the slave fetches hadoop
> executor by simply doing 'hadoop fs -copyToLocal <executor uri> <executor
> sandbox>'.****
>
> ****
>
> HTH,****
>
> ****
>
> On Tue, Aug 6, 2013 at 11:14 PM, Johnas, Nalini <njohnas@ebay.com> wrote:=
*
> ***
>
> ****
>
> Hi Vinod,****
>
> ****
>
> Yes. Exactly it gets lost as soon as it is lost and I am trying to figur=
e
> out why? There are no logs in the executor which makes it difficult to
> debug.****
>
> ****
>
> What are the potential root causes that could yield to the task getting
> lost as soon as it is launched? I could deep dive in that direction.****
>
> ****
>
> -Nalini****
>
> ****
>
> *From:* vinod@twitter.com [mailto:vinod@twitter.com] *On Behalf Of *Vinod
> Kone
> *Sent:* Sunday, August 04, 2013 10:48 PM
> *To:* user@mesos.apache.org
> *Subject:* Re: FW: Mesos slave not starting up****
>
> ****
>
> Was the syslog in one of the executor sandboxes? From the slave log you
> showed here, it looked like the executor went LOST as soon as it was
> launched (i.e., it never registered with the slave) but the syslog shows
> the executor came up?****
>
> ****
>
> The executor sandbox in this case would be *
> /tmp/mesos/slaves/201308040150-3892119818-5051-11035-0/frameworks/2013080=
40150-3892119818-5051-11035-0000/executors/executor_Task_Tracker_115/runs/d=
9094b15-540e-4370-a5b5-042b8c5ae6fa
> *****
>
> ****
>
> ** **
>
>
[Attachment #3 (text/html)]
<div dir="ltr"><div class="gmail_extra">We have pulled hadoop mesos framework out of \
mesos repo, to make it easy to for people to contribute and also to streamline its \
usage. This would be the only supported hadoop on mesos port going forward. Please \
give it a try and let us know how it works for you.<div class="gmail_extra">
<br><br><div class="gmail_quote">On Sun, Aug 11, 2013 at 11:24 PM, Johnas, Nalini \
<span dir="ltr"><<a href="mailto:njohnas@ebay.com" \
target="_blank">njohnas@ebay.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> \
<u></u></span></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks \
Vinod.<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> \
<u></u></span></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Sure will \
do. May I ask what’s different with this?<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> \
<u></u></span></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Nalini<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> \
<u></u></span></p><p class=""><b><span \
style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span \
style="font-size:10pt;font-family:Tahoma,sans-serif"> <a \
href="mailto:vinod@twitter.com" target="_blank">vinod@twitter.com</a> [mailto:<a \
href="mailto:vinod@twitter.com" target="_blank">vinod@twitter.com</a>] <b>On Behalf \
Of </b>Vinod Kone<br>
<b>Sent:</b> Sunday, August 11, 2013 8:35 PM<br><b>To:</b> <a \
href="mailto:user@mesos.apache.org" \
target="_blank">user@mesos.apache.org</a></span></p><div><div \
class="h5"><br><b>Subject:</b> Re: Mesos slave not starting up<u></u><u></u></div>
</div><p></p><div><div class="h5"><p class=""><u></u> <u></u></p><div><p class="">Can \
you try our new instructions at <a href="https://github.com/mesos/hadoop" \
target="_blank">https://github.com/mesos/hadoop</a> ?<u></u><u></u></p>
</div><div><p class="" style="margin-bottom:12pt"><u></u> <u></u></p><div><p \
class="">On Sun, Aug 11, 2013 at 7:19 PM, Johnas, Nalini <<a \
href="mailto:njohnas@ebay.com" target="_blank">njohnas@ebay.com</a>> \
wrote:<u></u><u></u></p>
<div><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi \
Vinod,</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I \
tried everything suggested, still running into the same problem with TASK LOST and \
there is no executor logs created.</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">One quick \
question, Is there any restriction with the Hadoop location , does it need to be \
under mesos build directory.</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Here’s \
where I have these installed.</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Mesos \
build is under : /home/njohnas_dev/mesos-testing/build</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hadoop \
home is under : /home/njohnas_dev/mesos-testing/hadoop</span><u></u><u></u></p><p \
class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Also \
I don’t mind driving up to a Starbucks closer to you, if you can spare like 1/2 hour \
with your busy schedule to go over my setup and help resolve this issue. (or) open to \
other suggestions as well. Let me know.</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Nalini</span><u></u><u></u></p><p \
class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p>
<div><div style="border-style:solid none \
none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in"><p \
class=""><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span \
style="font-size:10pt;font-family:Tahoma,sans-serif"> Johnas, Nalini [mailto:<a \
href="mailto:njohnas@ebay.com" target="_blank">njohnas@ebay.com</a>] <br>
<b>Sent:</b> Tuesday, August 06, 2013 11:59 PM<br><b>To:</b> <<a \
href="mailto:user@mesos.apache.org" \
target="_blank">user@mesos.apache.org</a>><br><b>Cc:</b> <a \
href="mailto:user@mesos.apache.org" target="_blank">user@mesos.apache.org</a><br>
<b>Subject:</b> Re: Mesos slave not starting \
up</span><u></u><u></u></p></div></div><div><p class=""> <u></u><u></u></p><div><p \
class="">Thanks Vinod that's helpful. I suspect it could be the hadoop path. Let \
me give this a try.<u></u><u></u></p>
</div><div><p class=""> <u></u><u></u></p></div><div><p class="">Nalini<br><br>Sent \
from my iPad<u></u><u></u></p></div><div><p class="" \
style="margin-bottom:12pt"><br>On Aug 6, 2013, at 11:52 PM, "Vinod Kone" \
<<a href="mailto:vinodkone@gmail.com" target="_blank">vinodkone@gmail.com</a>> \
wrote:<u></u><u></u></p>
</div><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><p class="">An \
executor terminated as soon as it's launched is indicative of slave being unable \
to fetch/launch the executor.<u></u><u></u></p><div><p class="">
<u></u><u></u></p></div><div><p class="">In the case of hadoop framework, If your \
executor sandbox doesn't have a hadoop.tar.gz or hadoop directory, that means the \
slave is unable to fetch the executor. It likely means the hdfs url for the executor \
specified in mapred-site.xml is wrong or inaccessible to the slave. \
<u></u><u></u></p>
</div><div><p class=""> <u></u><u></u></p></div><div><p class="">Also ensure that \
'hadoop' command is in the PATH of the slave (or specified via --hadoop_home \
slave flag), because the slave fetches hadoop executor by simply doing 'hadoop fs \
-copyToLocal <executor uri> <executor sandbox>'.<u></u><u></u></p>
</div><div><p class=""> <u></u><u></u></p></div><div><p \
class="">HTH,<u></u><u></u></p></div></div><div><p class="" \
style="margin-bottom:12pt"> <u></u><u></u></p><div><p class="">On Tue, Aug 6, 2013 at \
11:14 PM, Johnas, Nalini <<a href="mailto:njohnas@ebay.com" \
target="_blank">njohnas@ebay.com</a>> wrote:<u></u><u></u></p>
<div><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi \
Vinod,</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Yes. \
Exactly it gets lost as soon as it is lost and I am trying to figure out why? There \
are no logs in the executor which makes it difficult to \
debug.</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">What are \
the potential root causes that could yield to the task getting lost as soon as it is \
launched? I could deep dive in that direction.</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Nalini</span><u></u><u></u></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> \
</span><u></u><u></u></p><p class=""><b><span \
style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span \
style="font-size:10pt;font-family:Tahoma,sans-serif"> <a \
href="mailto:vinod@twitter.com" target="_blank">vinod@twitter.com</a> [mailto:<a \
href="mailto:vinod@twitter.com" target="_blank">vinod@twitter.com</a>] <b>On Behalf \
Of </b>Vinod Kone<br>
<b>Sent:</b> Sunday, August 04, 2013 10:48 PM<br><b>To:</b> <a \
href="mailto:user@mesos.apache.org" \
target="_blank">user@mesos.apache.org</a><br><b>Subject:</b> Re: FW: Mesos slave not \
starting up</span><u></u><u></u></p> <div>
<p class=""> <u></u><u></u></p><div><p class="">Was the syslog in one of the executor \
sandboxes? From the slave log you showed here, it looked like the executor went LOST \
as soon as it was launched (i.e., it never registered with the slave) but the syslog \
shows the executor came up?<u></u><u></u></p>
<div><p class=""> <u></u><u></u></p></div><div><p class="">The executor sandbox in \
this case would be <i><span \
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">/tmp/mesos/ \
slaves/201308040150-3892119818-5051-11035-0/frameworks/201308040150-3892119818-5051-11 \
035-0000/executors/executor_Task_Tracker_115/runs/d9094b15-540e-4370-a5b5-042b8c5ae6fa</span></i><u></u><u></u></p>
</div></div></div></div></div><p class=""> \
<u></u><u></u></p></div></blockquote></div></div></div><p class=""><u></u> \
<u></u></p></div><div><br></div></div></div></div></blockquote></div></div></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic