[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: spark on mesos with docker issue
From: Rad Gruchalski <radek () gruchalski ! com>
Date: 2015-11-10 22:47:33
Message-ID: 782D2F037A36453C923085241BE9CDC5 () gruchalski ! com
[Download RAW message or body]
Stavros,
As mentioned a couple of weeks ago: https://issues.apache.org/jira/browse/SPARK-11638
Happy to answer any questions.
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
\
(mailto:radek@gruchalski.com) de.linkedin.com/in/radgruchalski/ \
(http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential \
and/or legally privileged. If it has come to you in error you must take no action \
based on it, nor must you copy or show it to anyone; please delete/destroy and inform \
the sender immediately.
On Thursday, 22 October 2015 at 22:45, Stavros Kontopoulos wrote:
> Thnx Rad, sounds pretty cool :). Elizabeth one note for the jira ticket i do not \
> run the cluster with zookeeper, i use mesos master in stand alone mode... i guess \
> it makes no difference right?
> On Thu, Oct 22, 2015 at 10:38 PM, Rad Gruchalski <radek@gruchalski.com \
> (mailto:radek@gruchalski.com)> wrote:
> > There are 2 things:
> >
> > - Akka remote in 2.3.x does not support advertising hostname / port different to \
> > what it binds to
> > - All other services: file server, broadcast server, repl class server do not \
> > support advertising hostnames / ports different than what they bind to
> > Just to expand on the previous one, we are in the process of contributing the \
> > following bits:
> > - akka-remote bind-hostname and bind-port backport to akka 2.3.x (not the \
> > typesafe closed support implementation, our own implementation)
> > - spark patches for spark 1.4.0+ which enable running Spark on Mesos in Docker \
> > Bridge networking
> > Unfortunately, due to the nature of how my employer operates I can't share the \
> > code yet. We are working with our legal team to make these available asap.
> > We do run this stuff in production.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Kind regards,
> > Radek Gruchalski
> >
radek@gruchalski.com (mailto:radek@gruchalski.com)
\
> > (mailto:radek@gruchalski.com) de.linkedin.com/in/radgruchalski/ \
> > (http://de.linkedin.com/in/radgruchalski/)
> > Confidentiality:
> > This communication is intended for the above-named person and may be confidential \
> > and/or legally privileged. If it has come to you in error you must take no action \
> > based on it, nor must you copy or show it to anyone; please delete/destroy and \
> > inform the sender immediately.
> >
> >
> > On Thursday, 22 October 2015 at 22:28, Iulian Dragoș wrote:
> >
> > >
> > >
> > > On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <radek@gruchalski.com \
> > > (mailto:radek@gruchalski.com)> wrote:
> > > > Stavros,
> > > >
> > > > Spark does not support this. I am currently in the process of submitting \
> > > > patches for it however it first has to pass through the legal team at the \
> > > > company I work for.
> > >
> > > What exactly is missing in Spark?
> > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > > Radek Gruchalski
> > > >
radek@gruchalski.com (mailto:radek@gruchalski.com)
\
> > > > (mailto:radek@gruchalski.com) de.linkedin.com/in/radgruchalski/ \
> > > > (http://de.linkedin.com/in/radgruchalski/)
> > > > Confidentiality:
> > > > This communication is intended for the above-named person and may be \
> > > > confidential and/or legally privileged. If it has come to you in error you \
> > > > must take no action based on it, nor must you copy or show it to anyone; \
> > > > please delete/destroy and inform the sender immediately.
> > > >
> > > >
> > > > On Thursday, 22 October 2015 at 21:08, Stavros Kontopoulos wrote:
> > > >
> > > > > Bridge... with the latest mesos library vesion 0.25...
> > > > >
> > > > > On Thu, Oct 22, 2015 at 9:07 PM, Elizabeth Lingg <elizabeth@mesosphere.io \
> > > > > (mailto:elizabeth@mesosphere.io)> wrote:
> > > > > > Are you using Bridge or Host Networking?
> > > > > >
> > > > > > -Elizabeth
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 22, 2015 at 12:02 PM, Stavros Kontopoulos \
> > > > > > <st.kontopoulos@gmail.com (mailto:st.kontopoulos@gmail.com)> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Im using spark on mesos on docker. I have linked my slaves to the \
> > > > > > > master and a spark repl works fine inside the master container.
> > > > > > >
> > > > > > > If i try to crate the same spark repl form the host i get stuck at the \
> > > > > > > point when the framework tries to register to the mesos master (here \
> > > > > > > the framework is the spark repl itself). I can ping the container from \
> > > > > > > my host and vice versa. So networking its not the problem. What i \
> > > > > > > noticed form the logs is that mesos does not resolve the correct ip:
> > > > > > > Framework failover timeout, removing framework \
> > > > > > > b3605c33-f573-4d40-806f-b9b0abee2e32-0012 (Spark shell) at \
> > > > > > > scheduler-b6dbe4f8-4760-4cbe-bb5e-a991e35aba1b@127.0.1.1:40186 \
> > > > > > > (http://scheduler-b6dbe4f8-4760-4cbe-bb5e-a991e35aba1b@127.0.1.1:40186) \
> > > > > > > docker0 interface is on 172.17.x.x and my host is one such ip so i \
> > > > > > > didnt expect there to see 127.0.1.1. I have tried several things like \
> > > > > > > spark.driver.host, SPARK_LOCAL_IP to be set correctly but with no \
> > > > > > > result... I suspect this is a mesos problem on docker...
> > > > > > >
> > > > > > > Thnx,
> > > > > > >
> > > > > > > S.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > --
> > > Iulian Dragos
> > >
> > > ------
> > > Reactive Apps on the JVM
> > > www.typesafe.com (http://www.typesafe.com)
> > >
> >
>
[Attachment #3 (text/html)]
<div>
Stavros,
</div><div><br></div><div>As mentioned a couple of weeks ago: <a \
href="https://issues.apache.org/jira/browse/SPARK-11638">https://issues.apache.org/jira/browse/SPARK-11638</a></div><div>Happy \
to answer any questions.</div><div>
<p style="font-family: Helvetica; font-size: 10pt; margin-bottom: 0cm;">
</p><p style="font-family: Helvetica; font-size: 10pt; margin-bottom: 0cm;">
</p><p style="margin-bottom: 0cm;">
</p><p style="margin-bottom: 0cm"><font color="#000000"><font face="Helvetica, \
sans-serif"><font style="font-size: 9pt">Kind regards,
<br>Radek
Gruchalski<br>
</font></font></font><font face="Helvetica, sans-serif"><font \
style="font-size: 9pt"><a \
href="mailto:radek@gruchalski.com">radek@gruchalski.com</a><a \
href="mailto:radek@gruchalski.com">
</a></font></font><font color="#0084d1"><font \
face="Helvetica, sans-serif"><font style="font-size: 9pt"><br></font></font></font><a \
href="http://de.linkedin.com/in/radgruchalski/"><font color="#0084d1"><font \
face="Helvetica, sans-serif"><font style="font-size: \
9pt">de.linkedin.com/in/radgruchalski/</font></font></font></a><font \
color="#000000"><font face="Helvetica, sans-serif"><font style="font-size: \
9pt"><br><br></font></font></font><font color="#878787"><font face="Helvetica, \
sans-serif"><font style="font-size: \
8pt"><b>Confidentiality:<br></b></font></font></font><font color="#878787"><font \
face="Helvetica, sans-serif"><font style="font-size: 8pt">This communication is \
intended for the above-named person and may be confidential and/or legally \
privileged.<br>If it has come to you in error you must take no action based on it, \
nor must you copy or show it to anyone; please delete/destroy and inform the sender
immediately.</font></font></font></p></div>
<p style="color: #A0A0A8;">On Thursday, 22 October 2015 at 22:45, \
Stavros Kontopoulos wrote:</p>
<blockquote type="cite" \
style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"> \
<span><div><div><div dir="ltr">Thnx Rad, sounds pretty cool :). Elizabeth one note \
for the jira ticket i do not run the cluster with zookeeper, i use mesos master in \
stand alone mode... i guess it makes no difference right?<br></div><div><br><div>On \
Thu, Oct 22, 2015 at 10:38 PM, Rad Gruchalski <span dir="ltr"><<a \
href="mailto:radek@gruchalski.com" \
target="_blank">radek@gruchalski.com</a>></span> wrote:<br><blockquote \
type="cite"><div> <div>
There are 2 things:
</div><div><br></div><div> - Akka remote in 2.3.x does not \
support advertising hostname / port different to what it binds to</div><div> - \
All other services: file server, broadcast server, repl class server do not support \
advertising hostnames / ports different than what they bind \
to</div><div><br></div><div>Just to expand on the previous one, we are in the process \
of contributing the following bits:</div> <div><div><br></div><div> - \
akka-remote bind-hostname and bind-port backport to akka 2.3.x (not the typesafe \
closed support implementation, our own implementation)</div><div> - spark \
patches for spark 1.4.0+ which enable running Spark on Mesos in Docker Bridge \
networking</div><div><br></div><div>Unfortunately, due to the nature of how my \
employer operates I can't share the code yet. We are working with our legal team to \
make these available asap.</div><div><br></div><div>We do run this stuff in \
production.</div><span>
<p style="font-family:Helvetica;font-size:10pt;margin-bottom:0cm">
</p><p style="font-family:Helvetica;font-size:10pt;margin-bottom:0cm">
</p><p style="margin-bottom:0cm">
</p><p style="margin-bottom:0cm"><font color="#000000"><font face="Helvetica, \
sans-serif"><font style="font-size:9pt">Kind regards,
<br>Radek
Gruchalski<br>
</font></font></font><font face="Helvetica, sans-serif"><font \
style="font-size:9pt"><a href="mailto:radek@gruchalski.com" \
target="_blank">radek@gruchalski.com</a><a href="mailto:radek@gruchalski.com" \
target="_blank">
</a></font></font><font color="#0084d1"><font face="Helvetica, \
sans-serif"><font style="font-size:9pt"><br></font></font></font><a \
href="http://de.linkedin.com/in/radgruchalski/" target="_blank"><font \
color="#0084d1"><font face="Helvetica, sans-serif"><font \
style="font-size:9pt">de.linkedin.com/in/radgruchalski/</font></font></font></a><font \
color="#000000"><font face="Helvetica, sans-serif"><font \
style="font-size:9pt"><br><br></font></font></font><font color="#878787"><font \
face="Helvetica, sans-serif"><font \
style="font-size:8pt"><b>Confidentiality:<br></b></font></font></font><font \
color="#878787"><font face="Helvetica, sans-serif"><font style="font-size:8pt">This \
communication is intended for the above-named person and may be confidential and/or \
legally privileged.<br>If it has come to you in error you must take no action based \
on it, nor must you copy or show it to anyone; please delete/destroy and inform the \
sender immediately.</font></font></font></p></span></div><div><div>
<p style="color:#a0a0a8">On Thursday, 22 October 2015 at 22:28, \
Iulian Dragoș wrote:</p><blockquote type="cite"><div> <span><div><div><div \
dir="ltr"><br><div><br><div>On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <span \
dir="ltr"><<a href="mailto:radek@gruchalski.com" \
target="_blank">radek@gruchalski.com</a>></span> wrote:<br><blockquote \
type="cite"><div> <div>
Stavros,
</div><div><br></div><div>Spark does not support this. I am currently \
in the process of submitting patches for it however it first has to pass through the \
legal team at the company I work \
for.</div></div></blockquote><div><br></div><div>What exactly is missing in \
Spark?</div><div> </div><blockquote type="cite"><div><div>
<p style="font-family:Helvetica;font-size:10pt;margin-bottom:0cm">
</p><p style="font-family:Helvetica;font-size:10pt;margin-bottom:0cm">
</p><p style="margin-bottom:0cm">
</p><p style="margin-bottom:0cm"><font color="#000000"><font face="Helvetica, \
sans-serif"><font style="font-size:9pt">Kind regards,
<br>Radek
Gruchalski<br>
</font></font></font><font face="Helvetica, sans-serif"><font \
style="font-size:9pt"><a href="mailto:radek@gruchalski.com" \
target="_blank">radek@gruchalski.com</a><a href="mailto:radek@gruchalski.com" \
target="_blank">
</a></font></font><font color="#0084d1"><font face="Helvetica, \
sans-serif"><font style="font-size:9pt"><br></font></font></font><a \
href="http://de.linkedin.com/in/radgruchalski/" target="_blank"><font \
color="#0084d1"><font face="Helvetica, sans-serif"><font \
style="font-size:9pt">de.linkedin.com/in/radgruchalski/</font></font></font></a><font \
color="#000000"><font face="Helvetica, sans-serif"><font \
style="font-size:9pt"><br><br></font></font></font><font color="#878787"><font \
face="Helvetica, sans-serif"><font \
style="font-size:8pt"><b>Confidentiality:<br></b></font></font></font><font \
color="#878787"><font face="Helvetica, sans-serif"><font style="font-size:8pt">This \
communication is intended for the above-named person and may be confidential and/or \
legally privileged.<br>If it has come to you in error you must take no action based \
on it, nor must you copy or show it to anyone; please delete/destroy and inform the \
sender immediately.</font></font></font></p></div><div><div>
<p style="color:#a0a0a8">On Thursday, 22 October 2015 at 21:08, \
Stavros Kontopoulos wrote:</p><blockquote type="cite"><div> <span><div><div><div \
dir="ltr">Bridge... with the latest mesos library vesion \
0.25...<br></div><div><br><div>On Thu, Oct 22, 2015 at 9:07 PM, Elizabeth Lingg <span \
dir="ltr"><<a href="mailto:elizabeth@mesosphere.io" \
target="_blank">elizabeth@mesosphere.io</a>></span> wrote:<br><blockquote \
type="cite"><div><div dir="ltr"><div>Are you using Bridge or Host \
Networking?<span><font color="#888888"><br></font></span></div><span><font \
color="#888888"><div><br></div><div>-Elizabeth</div><div><br></div><div><br></div></font></span></div><div><div><div><br><div>On \
Thu, Oct 22, 2015 at 12:02 PM, Stavros Kontopoulos <span dir="ltr"><<a \
href="mailto:st.kontopoulos@gmail.com" \
target="_blank">st.kontopoulos@gmail.com</a>></span> wrote:<br><blockquote \
type="cite"><div><div dir="ltr"><div><div><div><div>Hi,<br><br></div>Im using spark \
on mesos on docker. I have linked my slaves to the master and a<br></div>spark repl \
works fine inside the master container.<br><br></div>If i try to crate the same spark \
repl form the host i get stuck at the point when the framework tries to register to \
the mesos master (here the framework is the spark repl itself).<br></div><div>I can \
ping the container from my host and vice versa. So networking its not the \
problem.<br></div><div>What i noticed form the logs is that mesos does not resolve \
the correct ip:<br><br>Framework failover timeout, removing framework \
b3605c33-f573-4d40-806f-b9b0abee2e32-0012 (Spark shell) at <a \
href="http://scheduler-b6dbe4f8-4760-4cbe-bb5e-a991e35aba1b@127.0.1.1:40186" \
target="_blank">scheduler-b6dbe4f8-4760-4cbe-bb5e-a991e35aba1b@127.0.1.1:40186</a><br><br></div><div>docker0 \
interface is on 172.17.x.x and my host is one such ip so i didnt expect there to see \
<br></div><div>127.0.1.1. I have tried several things like spark.driver.host, \
SPARK_LOCAL_IP to be set correctly but with no result...<br></div><div>I suspect this \
is a mesos problem on \
docker...<br></div><div><br></div><div>Thnx,<br><br></div>S.<br></div> \
</div></blockquote></div><br></div> </div></div></div></blockquote></div><br></div>
</div></div></span>
</div></blockquote><div>
<br>
</div>
</div></div></div></blockquote></div><br><br \
clear="all"><div><br></div>-- <br><div><div dir="ltr"><br>--<br>Iulian \
Dragos<br><div><br></div><div>------<br>Reactive Apps on the JVM<br><a \
href="http://www.typesafe.com" \
target="_blank">www.typesafe.com</a></div><div><br></div></div></div> </div></div>
</div></div></span>
</div></blockquote><div>
<br>
</div>
</div></div></div></blockquote></div><br></div>
</div></div></span>
</blockquote>
<div>
<br>
</div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic