[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: spark.cores.max=20 has no affect in Mesos 0.28.1
From: Rodrick Brown <rodrick () orchard-app ! com>
Date: 2016-04-29 17:43:16
Message-ID: 9bucymrdq7agjth1jx6pomc6j-0 () mailer ! nylas ! com
[Download RAW message or body]
Yeah this is so Mesos can use multiple executors from different machines and
allow dynamic allocations to scale up to 20 CORES but limit the maximum amount
to 20 with spark.cores.max ?
Am I missing something?
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
On Apr 29 2016, at 12:39 pm, June Taylor <june@umn.edu> wrote:
> You have specified total executor cores of 500.
>
>
>
>
>
> Thanks,
>
> June Taylor
>
> System Administrator, Minnesota Population Center
>
> University of Minnesota
>
>
>
> On Fri, Apr 29, 2016 at 11:35 AM, Rodrick Brown <[rodrick@orchard-
app.com](mailto:rodrick@orchard-app.com)> wrote:
>
>> I'm running a spark app on Mesos in coarse grain mode with dynmaic
allocation after my task launches I see it still takes more CPU's than what I
specify in my config. Does the following parameter spark.cores.max work in
Mesos or is this a YARN only parameter? is their something equivalent for
Mesos?
>>
>>
>>
>> When I go to the framework tab in Mesos I see this task consuming 30 CPU's
instead of 20 I limited it to.
>>
>>
>>
>> /usr/bin/timeout 0 /opt/spark-1.6.1/bin/spark-submit \
>>
>> \--master "mesos://zk://[prod-zookeeper-1.aws.xx.com:2181](http://prod-
zookeeper-1.aws.xx.com:2181),[prod-zookeeper-2.aws.xx.com:2181](http://prod-
zookeeper-2.aws.xx.com:2181),[prod-zookeeper-3.aws.xx.com:2181/mesos](http
://prod-zookeeper-3.aws.xx.com:2181/mesos)" \
>>
>> \--conf spark.ui.port1354 \
>>
>> \--conf spark.mesos.coarse=true \
>>
>> \--conf spark.cores.max \
>>
>> \--conf spark.mesos.constraints="rack:spark" \
>>
>> \--conf spark.sql.tungsten.enabled=true \
>>
>> \--conf spark.shuffle.service.enabled=true \
>>
>> \--conf spark.dynamicAllocation.enabled=true \
>>
>> \--conf spark.mesos.executor.memoryOverhead06 \
>>
>> \--class com.orchard.sparkjob.library.kafka.KafkaDirectConsumer \
>>
>> \--total-executor-cores 500 \
>>
>> \--driver-memory 2867M \
>>
>> \--executor-memory 4014M \
>>
>> \--jars /data/orchard/etc/config/es-services.jar \
>>
>> /data/orchard/jars/spark-job-library-
3e30539922ff540f4632d1d0745501c48300b89b-assembled.jar
>>
>>
>>
>> \--
>>
>> **Rodrick Brown** / Systems Engineer
>>
>> +1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
>>
>> **Orchard Platform**
>>
>> 101 5th Avenue, 4th Floor, New York, NY 10003
>>
>> [http://www.orchardplatform.com](http://www.orchardplatform.com/)
>>
>> [Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
>>
>>
>>
>> **NOTICE TO RECIPIENTS**: This communication is confidential and intended
for the use of the addressee only. If you are not an intended recipient of
this communication, please delete it immediately and notify the sender by
return email. Unauthorized reading, dissemination, distribution or copying of
this communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase any
loan, security or any other financial product or instrument, nor is it an
offer to sell or a solicitation of an indication of interest to purchase any
products or services to any persons who are prohibited from receiving such
information under applicable law. The contents of this communication may not
be accurate or complete and are subject to change without notice. As such,
Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes
no representation regarding the accuracy or completeness of the information
contained herein. The intended recipient is advised to consult its own
professional advisors, including those specializing in legal, tax and
accounting matters. Orchard does not provide legal, tax or accounting advice.
>
>
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
[Attachment #3 (unknown)]
<head></head><body><div>Yeah this is so Mesos can use multiple executors from \
different machines and allow dynamic allocations to scale up to 20 CORES but \
limit the maximum amount to 20 with spark.cores.max=20?</div><div><br></div><div>Am I \
missing something? </div><br><!-- <signature> \
--><div><br></div>-- <p></p><p style="margin-bottom: 0px; font-stretch: normal; \
font-size: 10px; line-height: normal; font-family: Verdana; color: rgb(102, 102, \
102); -webkit-text-stroke-color: rgb(102, 102, 102);"><b>Rodrick \
Brown</b> / Systems Engineer </p><p style="margin-bottom: 0px; \
font-stretch: normal; font-size: 10px; line-height: normal; font-family: Verdana; \
color: rgb(0, 148, 94); -webkit-text-stroke-color: rgb(0, 148, 94);">+1 917 445 6839 \
/ <a href="mailto:charlie@orchardplatform.com">rodrick@orchardplatform.com</a></p><p \
style="margin-bottom: 0px; font-stretch: normal; font-size: 10px; line-height: \
normal; font-family: Verdana; color: rgb(102, 102, 102); -webkit-text-stroke-color: \
rgb(102, 102, 102);"><b>Orchard Platform</b> </p><p style="margin-bottom: 0px; \
font-stretch: normal; font-size: 10px; line-height: normal; font-family: Verdana; \
color: rgb(102, 102, 102); -webkit-text-stroke-color: rgb(102, 102, 102);">101 5th \
Avenue, 4th Floor, New York, NY 10003 </p><p style="margin-bottom: 0px; \
font-stretch: normal; font-size: 10px; line-height: normal; font-family: Verdana; \
color: rgb(0, 148, 94); -webkit-text-stroke-color: rgb(0, 148, 94);"><a \
href="http://www.orchardplatform.com/">http://www.orchardplatform.com</a></p><p \
style="margin-bottom: 0px; font-stretch: normal; font-size: 10px; line-height: \
normal; font-family: Verdana; color: rgb(102, 102, 102); -webkit-text-stroke-color: \
rgb(102, 102, 102);"><a href="http://www.orchardplatform.com/blog/">Orchard \
Blog</a> | <a \
href="http://www.meetup.com/Peer-to-Peer-Lending-P2P/">Marketplace Lending \
Meetup</a></p><br><!-- </signature> --><div class="gmail_quote"> On Apr 29 2016, at \
12:39 pm, June Taylor <june@umn.edu> wrote: <br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex;"> <div dir="ltr">You have specified total executor cores of \
500.</div><div><br><div><div><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><br></div><div dir="ltr">Thanks,</div><div \
dir="ltr">June Taylor</div></div><div>System Administrator, Minnesota Population \
Center</div><div>University of \
Minnesota</div></div></div></div></div></div></div></div> <br><div>On Fri, Apr 29, \
2016 at 11:35 AM, Rodrick Brown <span dir="ltr"><<a \
href="mailto:rodrick@orchard-app.com" \
target="_blank">rodrick@orchard-app.com</a>></span> wrote:<br><blockquote \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>I'm \
running a spark app on Mesos in coarse grain mode \
with dynmaic allocation after my task launches I see it still takes more \
CPU's than what I specify in my config. Does the following parameter \
spark.cores.max=20 work in Mesos or is this a YARN only parameter? is their \
something equivalent for Mesos? </div><div><br></div><div>When I go to the \
framework tab in Mesos I see this task consuming 30 CPU's instead of 20 I \
limited it to. </div><div><br></div><div>/usr/bin/timeout 0 \
/opt/spark-1.6.1/bin/spark-submit \ </div><div>--master "mesos://zk://<a \
href="http://prod-zookeeper-1.aws.xx.com:2181" \
target="_blank">prod-zookeeper-1.aws.xx.com:2181</a>,<a \
href="http://prod-zookeeper-2.aws.xx.com:2181" \
target="_blank">prod-zookeeper-2.aws.xx.com:2181</a>,<a \
href="http://prod-zookeeper-3.aws.xx.com:2181/mesos" \
target="_blank">prod-zookeeper-3.aws.xx.com:2181/mesos</a>" \</div><div>--conf \
spark.ui.port=31354 \</div><div>--conf spark.mesos.coarse=true \</div><div>--conf \
spark.cores.max=20 \</div><div>--conf spark.mesos.constraints="rack:spark" \
\ </div><div>--conf spark.sql.tungsten.enabled=true \</div><div>--conf \
spark.shuffle.service.enabled=true \</div><div>--conf \
spark.dynamicAllocation.enabled=true \</div><div>--conf \
spark.mesos.executor.memoryOverhead=1606 \</div><div>--class \
com.orchard.sparkjob.library.kafka.KafkaDirectConsumer \
\</div><div>--total-executor-cores 500 \</div><div>--driver-memory 2867M \
\</div><div>--executor-memory 4014M \</div><div>--jars \
/data/orchard/etc/config/es-services.jar \
\</div><div>/data/orchard/jars/spark-job-library-3e30539922ff540f4632d1d0745501c48300b89b-assembled.jar \
</div><div><br></div>-- <p></p><p \
style="margin-bottom:0px;font-stretch:normal;font-size:10px;line-height:normal;font-family:Verdana;color:rgb(102,102,102)"><b>Rodrick \
Brown</b> / Systems Engineer </p><p \
style="margin-bottom:0px;font-stretch:normal;font-size:10px;line-height:normal;font-family:Verdana;color:rgb(0,148,94)"><a \
value="+19174456839" target="_blank">+1 917 445 6839</a> / <a \
href="mailto:charlie@orchardplatform.com" \
target="_blank">rodrick@orchardplatform.com</a></p><p \
style="margin-bottom:0px;font-stretch:normal;font-size:10px;line-height:normal;font-family:Verdana;color:rgb(102,102,102)"><b>Orchard \
Platform</b> </p><p \
style="margin-bottom:0px;font-stretch:normal;font-size:10px;line-height:normal;font-family:Verdana;color:rgb(102,102,102)">101 \
5th Avenue, 4th Floor, New York, NY 10003 </p><p \
style="margin-bottom:0px;font-stretch:normal;font-size:10px;line-height:normal;font-family:Verdana;color:rgb(0,148,94)"><a \
href="http://www.orchardplatform.com/" \
target="_blank">http://www.orchardplatform.com</a></p><p \
style="margin-bottom:0px;font-stretch:normal;font-size:10px;line-height:normal;font-family:Verdana;color:rgb(102,102,102)"><a \
href="http://www.orchardplatform.com/blog/" target="_blank">Orchard \
Blog</a> | <a href="http://www.meetup.com/Peer-to-Peer-Lending-P2P/" \
target="_blank">Marketplace Lending Meetup</a></p></div> <br>
<div><b>NOTICE TO RECIPIENTS</b>: This communication is confidential and intended for \
the use of the addressee only. If you are not an intended recipient of this \
communication, please delete it immediately and notify the sender by return \
email. Unauthorized reading, dissemination, distribution or copying of this \
communication is prohibited. This communication does not constitute an offer to \
sell or a solicitation of an indication of interest to purchase any loan, security or \
any other financial product or instrument, nor is it an offer to sell or a \
solicitation of an indication of interest to purchase any products or services \
to any persons who are prohibited from receiving such information under applicable \
law. The contents of this communication may not be accurate or complete and are \
subject to change without notice. As such, Orchard App, Inc. (including its \
subsidiaries and affiliates, "Orchard") makes no representation regarding \
the accuracy or completeness of the information contained herein. The intended \
recipient is advised to consult its own professional advisors, including those \
specializing in legal, tax and accounting matters. Orchard does not provide \
legal, tax or accounting advice.</div></blockquote></div><br></div> </blockquote>
</div></body>
<br>
<div><font size="1"><font><b>NOTICE TO RECIPIENTS</b>: This communication is \
confidential and intended for the use of the addressee only. If you are not an \
intended recipient of this communication, please delete it </font><font>immediately \
and notify the sender by return email. Unauthorized reading, dissemination, \
distribution or copying of this communication is prohibited. This communication does \
not </font>constitute an offer to sell or a solicitation of an indication of interest \
to purchase any loan, security or any other financial product or instrument, nor is \
it an offer to sell or a solicitation of an indication of interest to purchase any \
products or services to any persons who are prohibited from receiving such \
information under applicable law. The contents of this communication may not be \
accurate or complete and are subject to change without notice. As such, Orchard App, \
Inc. (including its subsidiaries and affiliates, "Orchard") makes no \
representation regarding the <font>accuracy or completeness of the information \
contained herein. The intended recipient is advised to consult its own professional \
advisors, including those specializing in legal, tax and accounting </font>matters. \
Orchard does not provide legal, tax or accounting advice.</font></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic