[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mesos-user
Subject:    Re: 0.21.0-pre & Spark latest
From:       Timothy Chen <tnachen () gmail ! com>
Date:       2014-10-28 16:09:37
Message-ID: CAFx0iW_qA+zjhZoaXS0b3CkVcW0sQkoirFAiXLw4R0Vf_EwHog () mail ! gmail ! com
[Download RAW message or body]

Hi RJ,

I see, are you or the team on working on this problem already? If not
I'd like to take a look as well.

Tim

On Tue, Oct 28, 2014 at 8:47 AM, RJ Nowling <rnowling@redhat.com> wrote:
> Hi Tim,
> 
> The integration test is simply to open the spark shell (Spark 1.1.0) using mesos \
> 0.21 in coarse-grained mode.  We didn't even have to run any commands. 
> RJ
> 
> ----- Original Message -----
> > From: "Timothy Chen" <tnachen@gmail.com>
> > To: dev@mesos.apache.org
> > Cc: user@mesos.apache.org, "RJ Nowling" <rnowling@redhat.com>, "Erik Erlandson" \
> >                 <eje@redhat.com>
> > Sent: Tuesday, October 28, 2014 11:40:19 AM
> > Subject: Re: 0.21.0-pre & Spark latest
> > 
> > Hi Tim,
> > 
> > Thanks for doing the integration tests, that's something that I wanted to do
> > but never got to yet.
> > 
> > I have great interest ensuring spark and mesos work, and I know Brenden as
> > well does.
> > 
> > I have been tracking these spark mesos problems with spark jira and labeling
> > them mesos. Can you create these bugs on jira and we can dig more on each
> > one?
> > 
> > Also is this integration test automated?
> > 
> > Thanks!
> > 
> > Tim
> > 
> > > On Oct 28, 2014, at 8:03 AM, Tim St Clair <tstclair@redhat.com> wrote:
> > > 
> > > inline
> > > 
> > > ----- Original Message -----
> > > 
> > > > From: "Brenden Matthews" <brenden.matthews@airbedandbreakfast.com>
> > > > To: user@mesos.apache.org
> > > > Cc: "mesos-devel" <dev@mesos.apache.org>, "RJ Nowling"
> > > > <rnowling@redhat.com>,
> > > > "Erik Erlandson" <eje@redhat.com>
> > > > Sent: Tuesday, October 28, 2014 9:51:58 AM
> > > > Subject: Re: 0.21.0-pre & Spark latest
> > > 
> > > > Since we've recently adopted Spark, I'll second Tim's comment. We had an
> > > > issue with 0.20.1 that was possibly related to Spark[1], so it's important
> > > > for us to get this stuff fixed in 0.21.0.
> > > 
> > > > Tim, can you elaborate on the issues you saw? Have you tested with my
> > > > recent
> > > > Spark patches[2][3]?
> > > 
> > > We are building against Spark 1.1.0 unpatched:
> > > - Fine grained mode appears broken.
> > > 
> > > - Course grained mode appears to work via normal runs, but crashes in the
> > > REPL.
> > > http://fpaste.org/145782/14506564/
> > > 
> > > > [1]: https://issues.apache.org/jira/browse/MESOS-1973
> > > > [2]: https://github.com/apache/spark/pull/2401
> > > > [3]: https://github.com/apache/spark/pull/2453
> > > 
> > > > On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com >
> > > > wrote:
> > > 
> > > > > Folks -
> > > 
> > > > > We have some automated tests that run the latest Mesos against the latest
> > > > > Spark, and we've run across a series of issues in both fine, and course
> > > > > grained mode that I believe stem from a series of changes in the 0.21
> > > > > cycle.
> > > 
> > > > > I'm not certain if anyone "owns" this integration, but we should probably
> > > > > ensure it's fixed before we push out 0.21.
> > > 
> > > > > --
> > > > 
> > > > > Cheers,
> > > > 
> > > > > Timothy St. Clair
> > > > 
> > > > > Red Hat Inc.
> > > 
> > > --
> > > Cheers,
> > > Timothy St. Clair
> > > Red Hat Inc.
> > 


[prev in list] [next in list] [prev in thread] [next in thread]