[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: Spark Bootstrapping on Mesos
From: Stephen Carman <scarman () coldlight ! com>
Date: 2015-05-13 18:13:07
Message-ID: 26638C6A-7004-400A-B282-8DEBB3F2504A () coldlight ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Okay my apologies perhaps I'm just not explaining this well.
So we have an application, that creates a spark context and adds all the necessary \
jars, the job is submitted to the cluster and as it runs the code it attempts to pull \
the data necessary for the job from s3. We use our own version of Presto's s3 \
filesystem to be able to read and write the data from s3, but that needs to be \
initialized with our keys and the s3 bucket before the code can understand how to \
handle that file. All the Jars and everything get to the slave fine, so it has no \
problems with ClassNotFound, it just doesn't seem to be initilizing this s3 file \
system, so when it tries to download the data from s3 it fails saying it doesn't know \
how to handle an s3 URI.
Hopefully that makes sense, any insight?
Thanks,
Steve
On May 13, 2015, at 2:03 PM, Tim Chen <tim@mesosphere.io<mailto:tim@mesosphere.io>> \
wrote:
Hi Stephen,
I'm not quite sure what you mean by bootstrapping classes, do you have some \
particular examples?
Usually to run any user jar you just need it to be reachable by your slaves so it can \
be either S3 or any accessible place, then you just provide your jar url when you run \
spark-submit.
Tim
On Wed, May 13, 2015 at 8:09 AM, Stephen Carman \
<scarman@coldlight.com<mailto:scarman@coldlight.com>> wrote: Hi,
We have a small mesos cluster and we'd like to be able to initialize some of our \
classes, mostly we have a vfs we setup to be allow our code to access S3, but there \
doesn't seem to be any readily obvious way to bootstrap these kind of classes so that \
they have the properly initialized configuration they need to operate.
Is there some accepted way to accomplish this?
thanks,
Steve
This e-mail is intended solely for the above-mentioned recipient and it may contain \
confidential or privileged information. If you have received it in error, please \
notify us immediately and delete the e-mail. You must not copy, distribute, disclose \
or take any action in reliance on it. In addition, the contents of an attachment to \
this e-mail may contain software viruses which could damage your own computer system. \
While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this \
risk, we cannot accept liability for any damage which you sustain as a result of \
software viruses. You should perform your own virus checks before opening the \
attachment.
This e-mail is intended solely for the above-mentioned recipient and it may contain \
confidential or privileged information. If you have received it in error, please \
notify us immediately and delete the e-mail. You must not copy, distribute, disclose \
or take any action in reliance on it. In addition, the contents of an attachment to \
this e-mail may contain software viruses which could damage your own computer system. \
While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this \
risk, we cannot accept liability for any damage which you sustain as a result of \
software viruses. You should perform your own virus checks before opening the \
attachment.
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space;" class=""> Okay my apologies perhaps I'm just not explaining this \
well. <div class=""><br class="">
</div>
<div class="">So we have an application, that creates a spark context and adds all \
the necessary jars, the job is submitted to the cluster</div> <div class="">and as it \
runs the code it attempts to pull the data necessary for the job from s3. We use our \
own version of Presto's s3 filesystem</div> <div class="">to be able to read and \
write the data from s3, but that needs to be initialized with our keys and the s3 \
bucket before the code can understand</div> <div class="">how to handle that file. \
All the Jars and everything get to the slave fine, so it has no problems with \
ClassNotFound, it just doesn't seem to be initilizing</div> <div class="">this s3 \
file system, so when it tries to download the data from s3 it fails saying it doesn't \
know how to handle an s3 URI.</div> <div class=""><br class="">
</div>
<div class="">Hopefully that makes sense, any insight?</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Steve</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 13, 2015, at 2:03 PM, Tim Chen <<a \
href="mailto:tim@mesosphere.io" class="">tim@mesosphere.io</a>> wrote:</div> <br \
class="Apple-interchange-newline"> <div class="">
<div dir="ltr" class="">Hi Stephen,
<div class=""><br class="">
</div>
<div class="">I'm not quite sure what you mean by bootstrapping classes, do you have \
some particular examples?</div> <div class=""><br class="">
</div>
<div class="">Usually to run any user jar you just need it to be reachable by your \
slaves so it can be either S3 or any accessible place, then you just provide your jar \
url when you run spark-submit.</div> <div class=""><br class="">
</div>
<div class="">Tim</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, May 13, 2015 at 8:09 AM, Stephen Carman <span \
dir="ltr" class=""> <<a href="mailto:scarman@coldlight.com" target="_blank" \
class="">scarman@coldlight.com</a>></span> wrote:<br class=""> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> Hi,<br class="">
<br class="">
We have a small mesos cluster and we'd like to be able to initialize some of our \
classes, mostly we have a vfs we setup to be allow our code<br class=""> to access \
S3, but there doesn't seem to be any readily obvious way to bootstrap these kind of \
classes so that they have the properly initialized configuration they need<br \
class=""> to operate.<br class="">
<br class="">
Is there some accepted way to accomplish this?<br class="">
<br class="">
<br class="">
thanks,<br class="">
Steve<br class="">
This e-mail is intended solely for the above-mentioned recipient and it may contain \
confidential or privileged information. If you have received it in error, please \
notify us immediately and delete the e-mail. You must not copy, distribute, disclose \
or take any action in reliance on it. In addition, the contents of an attachment to \
this e-mail may contain software viruses which could damage your own computer system. \
While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this \
risk, we cannot accept liability for any damage which you sustain as a result of \
software viruses. You should perform your own virus checks before opening the \
attachment.<br class=""> </blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
This e-mail is intended solely for the above-mentioned recipient and it may contain \
confidential or privileged information. If you have received it in error, please \
notify us immediately and delete the e-mail. You must not copy, distribute, disclose \
or take any action in reliance on it. In addition, the contents of an attachment to \
this e-mail may contain software viruses which could damage your own computer system. \
While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this \
risk, we cannot accept liability for any damage which you sustain as a result of \
software viruses. You should perform your own virus checks before opening the \
attachment. </body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic