[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: CDH5.2.3 on mesos
From: Stratos Dimopoulos <stratos.dimopoulos () gmail ! com>
Date: 2014-10-31 21:29:23
Message-ID: CAGWuDaGwraT=k2Z6=HVoKnWYCGFzzPK2Y2Go5x3pkB21bNUrUw () mail ! gmail ! com
[Download RAW message or body]
High Availability you mean? Why not Zookeeper?
On Thu, Oct 30, 2014 at 7:07 PM, Ankur Chauhan <ankur@malloc64.com> wrote:
> Thanks this is great. I actually ended up using the vanilla Hadoop
> distribution and it worked just fine. I will try out your tutorial. Side
> question, is there a solution around making the jobtracker ha?
>
> -- ankur
>
> Sent from my iPhone
>
> On Oct 30, 2014, at 5:05 PM, Stratos Dimopoulos <
> stratos.dimopoulos@gmail.com> wrote:
>
> Hi Ankur,
>
> I recently went through the process of installing Hadoop on Mesos over
> cdh5.1.2
>
> I created a post here - shouldn't be much different for your version:
> http://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/
>
> You can also find an other post about configuring CDH5.1.2 specifically to
> use with Mesos: useful:
> http://strat0sphere.wordpress.com/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/
>
> Have in mind that when using Mesos you don't need to start the jobtracker.
> Mesos will do this for you.
> You also mentioned that you are trying to start hadoop as ubuntu user.
> This is not the right thing to do. Either add root to cloudera's root list
> or (recommended) use the root user (mapred? hdfs?) that your cloudera
> version considers as root - you have to check the documentation for this.
>
> Regarding the error you are seeing "Does not contain a valid host:port
> authority: local" - I've seen this error when my worker version was
> different than the jobtracker version (happened because I was using a
> hadoop-on-mesos tar file compiled with a different version than the one my
> cluster was using). To fix this you can do the obvious, which is making
> sure the installed version is the same with the one you ship to the
> executors through HDFS or you can hack this by adding the property hadoop.skip.worker.version.check
> to True - In the later case I wish you good luck... Neverhteless, I am not
> sure if this error can also appear in other cases.
>
> Hope this helps.
>
> Stratos
>
>
> On Tue, Oct 28, 2014 at 12:30 PM, Ankur Chauhan <ankur@malloc64.com>
> wrote:
>
>> Anyone else have something to add on this?
>> -- Ankur Chauhan
>>
>> On 28 Oct 2014, at 02:10, Ankur Chauhan <ankur@malloc64.com> wrote:
>>
>> Hi tom,
>>
>> I was basically following the readme. This gist has the list of commands
>> how i am setting up things
>> https://gist.github.com/ankurcha/a9504b0e423b1a40d756 so first of all if
>> possible if you could help me verify if my process of setting up core-site,
>> hdfs-site and mapred-site is correct. I was starting the node with
>> $ /opt/hadoop/bin/hadoop jobtracker
>>
>> There are two errors that i was working through. It seems that hadoop
>> doesn't like running as root (which is good) but despite starting the
>> process as ubuntu i kept getting
>>
>> Does not contain a valid host:port authority: local
>>
>> -- Ankur
>>
>> On 28 Oct 2014, at 01:57, Tom Arnfeld <tom@duedil.com> wrote:
>>
>> Hi Ankur,
>>
>> There aren't any getting started resources other than the documention
>> there as far as I know. Could you share your hadoop configuration and
>> perhaps a description of the problems you're having?
>>
>> Tom.
>>
>>
>>
>> On Tue, Oct 28, 2014 at 8:53 AM, Ankur Chauhan <ankur@malloc64.com>
>> wrote:
>>
>>> H,
>>>
>>> I was trying to setup mesos/hadoop with the latest CDH version (MR1) and
>>> it seems like the instructions are sort of out of date and I also tried the
>>> suggestions in https://github.com/mesos/hadoop/issues/25 but after 4
>>> hours of flailing around I am still kind of stuck :-/
>>>
>>> It seems like the configuration/installation instructions aren't
>>> complete and I am just too new to hadoop to figure out what's missing or
>>> going wrong. Does anyone know of a good resource I can use to get going?
>>>
>>> -- Ankur
>>>
>>
>>
>>
>>
>
[Attachment #3 (text/html)]
<div dir="ltr">High Availability you mean? Why not Zookeeper?</div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 30, 2014 at 7:07 PM, \
Ankur Chauhan <span dir="ltr"><<a href="mailto:ankur@malloc64.com" \
target="_blank">ankur@malloc64.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="auto"><div>Thanks this is great. I actually ended \
up using the vanilla Hadoop distribution and it worked just fine. I will try out your \
tutorial. Side question, is there a solution around making the jobtracker \
ha?</div><div><br></div><div>-- ankur <br><br>Sent from my iPhone</div><div><div \
class="h5"><div><br>On Oct 30, 2014, at 5:05 PM, Stratos Dimopoulos <<a \
href="mailto:stratos.dimopoulos@gmail.com" \
target="_blank">stratos.dimopoulos@gmail.com</a>> wrote:<br><br></div><blockquote \
type="cite"><div><div dir="ltr">Hi Ankur,<div><br></div><div>I recently went through \
the process of installing Hadoop on Mesos over cdh5.1.2</div><div><br></div><div>I \
created a post here - shouldn't be much different for your version: <a \
href="http://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/" \
target="_blank">http://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/</a></div><div><br></div><div>You \
can also find an other post about configuring CDH5.1.2 specifically to use with \
Mesos: useful: <a href="http://strat0sphere.wordpress.com/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/" \
target="_blank">http://strat0sphere.wordpress.com/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/</a></div><div><br></div><div>Have \
in mind that when using Mesos you don't need to start the jobtracker. Mesos will \
do this for you. </div><div>You also mentioned that you are trying to start hadoop \
as ubuntu user. This is not the right thing to do. Either add root to cloudera's \
root list or (recommended) use the root user (mapred? hdfs?) that your cloudera \
version considers as root - you have to check the documentation for \
this.</div><div><br></div><div>Regarding the error you are seeing "<span \
style="font-size:12.7272720336914px;color:rgb(51,51,51);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap">Does \
not contain a valid host:port authority: </span><span \
style="font-size:12.7272720336914px;color:rgb(0,134,179);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap">local</span>" \
- I've seen this error when my worker version was different than the jobtracker \
version (happened because I was using a hadoop-on-mesos tar file compiled with a \
different version than the one my cluster was using). To fix this you can do the \
obvious, which is making sure the installed version is the same with the one you ship \
to the executors through HDFS or you can hack this by adding the property <span \
style="font-family:Arial;font-size:13px;white-space:pre-wrap">hadoop.skip.worker.version.check \
to True - In the later case I wish you good luck... Neverhteless, I am not sure if \
this error can also appear in other cases.</span></div><div><br></div><div>Hope this \
helps.</div><div><br></div><div>Stratos</div><div><br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 28, 2014 at 12:30 PM, \
Ankur Chauhan <span dir="ltr"><<a href="mailto:ankur@malloc64.com" \
target="_blank">ankur@malloc64.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div style="word-wrap:break-word">Anyone else have something \
to add on this?<span><font color="#888888"><div>-- Ankur \
Chauhan</div></font></span><div><div><div><br><div><blockquote type="cite"><div>On 28 \
Oct 2014, at 02:10, Ankur Chauhan <<a href="mailto:ankur@malloc64.com" \
target="_blank">ankur@malloc64.com</a>> wrote:</div><br><div><div \
style="word-wrap:break-word">Hi tom,<div><br></div><div>I was basically following the \
readme. This gist has the list of commands how i am setting up things</div><div><a \
href="https://gist.github.com/ankurcha/a9504b0e423b1a40d756" \
target="_blank">https://gist.github.com/ankurcha/a9504b0e423b1a40d756</a> so first \
of all if possible if you could help me verify if my process of setting up core-site, \
hdfs-site and mapred-site is correct. I was starting the node with </div><div><span \
style="white-space:pre-wrap"> </span></div><div><span \
style="white-space:pre-wrap"> </span>$ /opt/hadoop/bin/hadoop \
jobtracker</div><div><br></div><div>There are two errors that i was working through. \
It seems that hadoop doesn't like running as root (which is good) but despite \
starting the process as ubuntu i kept getting </div><div><br></div><div><span \
style="color:rgb(51,51,51);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap;background-color:rgb(255,255,255)"> \
Does not contain a valid host:port authority: </span><span \
style="color:rgb(0,134,179);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap;background-color:rgb(255,255,255)">local</span></div><div><span \
style="color:rgb(0,134,179);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap;background-color:rgb(255,255,255)"><br></span></div><div><span \
style="color:rgb(0,134,179);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap;background-color:rgb(255,255,255)">-- \
Ankur</span></div><div><span \
style="color:rgb(0,134,179);font-family:Consolas,'Liberation \
Mono',Menlo,Courier,monospace;line-height:16.7999992370605px;white-space:pre-wrap;background-color:rgb(255,255,255)"><br></span></div><div><div><blockquote \
type="cite"><div>On 28 Oct 2014, at 01:57, Tom Arnfeld <<a \
href="mailto:tom@duedil.com" target="_blank">tom@duedil.com</a>> \
wrote:</div><br><div> <span>Hi Ankur,<div><br></div>
<div>There aren't any getting started resources other than the documention there \
as far as I know. Could you share your hadoop configuration and perhaps a description \
of the problems you're having?</div> <div><br></div>
<div>Tom.</div></span><div><br></div>
<br><br><div class="gmail_quote"><p>On Tue, Oct 28, 2014 at 8:53 AM, Ankur Chauhan \
<span dir="ltr"><<a href="mailto:ankur@malloc64.com" \
target="_blank">ankur@malloc64.com</a>></span> wrote:<br></p><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p>H,</p> <div><br></div>
<div>I was trying to setup mesos/hadoop with the latest CDH version (MR1) and it \
seems like the instructions are sort of out of date and I also tried the suggestions \
in <a href="https://github.com/mesos/hadoop/issues/25" \
target="_blank">https://github.com/mesos/hadoop/issues/25</a> but after 4 hours of \
flailing around I am still kind of stuck :-/</div> <div><br></div>
<div>It seems like the configuration/installation instructions aren't complete \
and I am just too new to hadoop to figure out what's missing or going wrong. Does \
anyone know of a good resource I can use to get going?</div> <div><br></div>
<div>-- Ankur</div>
</blockquote></div><br></div></blockquote></div><br></div></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div>
</div></blockquote></div></div></div></blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic