[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    YARN Fair Scheduler - maxResources limit for parent and child queues with inheritance
From:       David Fox <dafox7777777 () gmail ! com>
Date:       2015-09-29 15:29:51
Message-ID: CADT6aMQhunrkOq2huy5Cd3zmJLodRvxW7pmF=eGu2f3Y98m-wA () mail ! gmail ! com
[Download RAW message or body]

Hello,

I would like to know how it works with Fair Scheduler in YARN-hadoop. I'm
trying to configure parent queue with MaxResorces in allocations file, e.i.
fair-scheduler.xml. Then I want to create child queues. As documentation
about fair scheduler says "Queues can be arranged in a hierarchy to divide
resources and configured with weights to share the cluster in specific
proportions"
http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
I suppose if I configure parent queue with Max Resources, the child queues
will be limited to max of the parent resources with fair shares. Is my
undersatnding of queue hierarchy mechanism right?

Example to be more concrete:
{code}
<?xml version="1.0"?>
<allocations>
    <queue name="someParentQueue" type="parent">
        <maxResources>111000 mb,100 vcores, 100 disks</maxResources>
    </queue>
    <queuePlacementPolicy>
        <rule name="nestedUserQueue" create="true">
            <rule name="specified" create="false"/>
        </rule>
        <rule name="user" create="true"/>
    </queuePlacementPolicy>
</allocations>
{code}

The queues someParentQueue.user1 has different Max Resources in web UI. Is
it issue of web UI or intended behavoior of Fair Scheduler?

Thanks

[Attachment #3 (text/html)]

<div dir="ltr"><div><div><div><div><div><div>Hello,<br><br></div>I would like to know \
how it works with Fair Scheduler in YARN-hadoop. I&#39;m trying to configure parent \
queue with MaxResorces in allocations file, e.i. fair-scheduler.xml. Then I want to \
create child queues. As documentation about fair scheduler says &quot;Queues can be \
arranged in a hierarchy to divide resources and configured  with weights to share the \
cluster in specific proportions&quot; <a \
href="http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html" \
target="_blank">http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html</a><br></div>I \
suppose if I configure parent queue with Max Resources, the child queues will be \
limited to max of the parent resources with fair shares. Is my undersatnding of queue \
hierarchy mechanism right?<br><br>Example to be more concrete:<br></div>{code}<span \
style="font-style:italic"><br>&lt;?xml \
version=&quot;1.0&quot;?&gt;<br>&lt;allocations&gt;<br>       &lt;queue \
name=&quot;someParentQueue&quot; type=&quot;parent&quot;&gt;<br>               \
&lt;maxResources&gt;111000 mb,100 vcores, 100 disks&lt;/maxResources&gt;<br>       \
&lt;/queue&gt;<br>       &lt;queuePlacementPolicy&gt;<br>               &lt;rule \
name=&quot;nestedUserQueue&quot; create=&quot;true&quot;&gt;<br>                      \
&lt;rule name=&quot;specified&quot; create=&quot;false&quot;/&gt;<br>               \
&lt;/rule&gt;<br>               &lt;rule name=&quot;user&quot; \
create=&quot;true&quot;/&gt;<br>       \
&lt;/queuePlacementPolicy&gt;<br>&lt;/allocations&gt;<br></span></div>{code} \
<br><br></div>The queues someParentQueue.user1 has different Max Resources in web UI. \
Is it issue of web UI or intended behavoior of Fair \
Scheduler?<br><br></div>Thanks<br></div>



[prev in list] [next in list] [prev in thread] [next in thread]