[prev in list] [next in list] [prev in thread] [next in thread]
List: mesos-user
Subject: Re: Default executor grace period
From: Alex Rukletsov <alex () mesosphere ! com>
Date: 2017-04-25 14:52:27
Message-ID: CACzFoUvtaBfTwOF4FhiW8YJC9vYEjS1t8VLVj6HGiNoP37R+zw () mail ! gmail ! com
[Download RAW message or body]
Commented on the ticket.
On Tue, Jan 17, 2017 at 12:27 PM, Tomek Janiszewski <janiszt@gmail.com>
wrote:
> Created issue for this: https://issues.apache.org/jira/browse/MESOS-6933
>
> pon., 16 sty 2017 o 17:13 u=C5=BCytkownik Tomek Janiszewski <janiszt@gmai=
l.com>
> napisa=C5=82:
>
>> I looks like it's supported because executor prints grace period[1]. On
>> the other hand executor launches sh that launch command and shell execut=
es
>> faster then command after receiving SIGTERM. Causing process to be attac=
hed
>> to init and leaked. In my opinion default executor should not sent SIGTE=
RM
>> to sh but only to its children. This will allow proper escalation to
>> SIGKILL because sh will leave as long its children are alive.
>>
>> 1: https://github.com/apache/mesos/blob/c4667d6f1b49d30089e6cb5874b673
>> 7a9bd3f044/src/launcher/executor.cpp#L479-L480
>>
>> pon., 16 sty 2017 o 16:35 u=C5=BCytkownik haosdent <haosdent@gmail.com>
>> napisa=C5=82:
>>
>> It looks like default-executor have not yet handle
>> `--executor_shutdown_grace_period`=E3=80=82
>>
>> On Mon, Jan 16, 2017 at 7:41 PM, Tomek Janiszewski <janiszt@gmail.com>
>> wrote:
>>
>> Hi
>>
>> I tried to use grace period with default Mesos executor. I assumed it
>> works as follow:
>>
>> 1. Start command: sh -c "command ..."
>> 2. Sent SIGSTOP to process tree: sh, command
>> 3. Sent SIGTERM to process tree: sh, command
>> 4. Wait for processes to finish or grace period to elapse
>> 5. sh finish while command could be still running and attached to ini=
t
>> 6. Sent SIGKILL to process tree: command
>>
>> I notice that SIGKILL is not sent and executor finished when sh returns.
>> When Mesos is running with POSIX contenerizer this leads command to live
>> forever (if it ignores SIGTERM). When contenerizer is used command is
>> killed when it's container is destroyed.
>>
>> Is this desired behavior? How to use grace period with default executor?
>>
>> Thanks
>> Tomek
>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>>
[Attachment #3 (text/html)]
<div dir="ltr">Commented on the ticket.</div><div class="gmail_extra"><br><div \
class="gmail_quote">On Tue, Jan 17, 2017 at 12:27 PM, Tomek Janiszewski <span \
dir="ltr"><<a href="mailto:janiszt@gmail.com" \
target="_blank">janiszt@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Created issue for this: <a \
href="https://issues.apache.org/jira/browse/MESOS-6933" \
target="_blank">https://issues.apache.org/<wbr>jira/browse/MESOS-6933</a></div><div \
class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">pon., 16 \
sty 2017 o 17:13 użytkownik Tomek Janiszewski <<a \
href="mailto:janiszt@gmail.com" target="_blank">janiszt@gmail.com</a>> \
napisał:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" \
class="m_-8284271755943322836gmail_msg">I looks like it's supported because \
executor prints grace period[1]. On the other hand executor launches sh that launch \
command and shell executes faster then command after receiving SIGTERM. Causing \
process to be attached to init and leaked. In my opinion default executor should not \
sent SIGTERM to sh but only to its children. This will allow proper escalation to \
SIGKILL because sh will leave as long its children are alive.<div \
class="m_-8284271755943322836gmail_msg"><br \
class="m_-8284271755943322836gmail_msg"></div><div \
class="m_-8284271755943322836gmail_msg">1: <a \
href="https://github.com/apache/mesos/blob/c4667d6f1b49d30089e6cb5874b6737a9bd3f044/src/launcher/executor.cpp#L479-L480" \
class="m_-8284271755943322836gmail_msg" \
target="_blank">https://github.com/apache/<wbr>mesos/blob/<wbr>c4667d6f1b49d30089e6cb5 \
874b673<wbr>7a9bd3f044/src/launcher/<wbr>executor.cpp#L479-L480</a></div></div><br \
class="m_-8284271755943322836gmail_msg"><div class="gmail_quote \
m_-8284271755943322836gmail_msg"><div dir="ltr" \
class="m_-8284271755943322836gmail_msg">pon., 16 sty 2017 o 16:35 użytkownik \
haosdent <<a href="mailto:haosdent@gmail.com" \
class="m_-8284271755943322836gmail_msg" target="_blank">haosdent@gmail.com</a>> \
napisał:<br class="m_-8284271755943322836gmail_msg"></div><blockquote \
class="gmail_quote m_-8284271755943322836gmail_msg" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" \
class="m_-8284271755943322836gmail_msg">It looks like default-executor have not yet \
handle `--executor_shutdown_grace_<wbr>period`。</div><div class="gmail_extra \
m_-8284271755943322836gmail_msg"></div><div class="gmail_extra \
m_-8284271755943322836gmail_msg"><br class="m_-8284271755943322836gmail_msg"><div \
class="gmail_quote m_-8284271755943322836gmail_msg">On Mon, Jan 16, 2017 at 7:41 PM, \
Tomek Janiszewski <span dir="ltr" class="m_-8284271755943322836gmail_msg"><<a \
href="mailto:janiszt@gmail.com" class="m_-8284271755943322836gmail_msg" \
target="_blank">janiszt@gmail.com</a>></span> wrote:<br \
class="m_-8284271755943322836gmail_msg"><blockquote class="gmail_quote \
m_-8284271755943322836gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr" class="m_-8284271755943322836gmail_msg">Hi<br \
class="m_-8284271755943322836gmail_msg"><br class="m_-8284271755943322836gmail_msg">I \
tried to use grace period with default Mesos executor. I assumed it works as \
follow:<br class="m_-8284271755943322836gmail_msg"><ol \
class="m_-8284271755943322836gmail_msg"><li \
class="m_-8284271755943322836gmail_msg">Start command: sh -c "command \
..."<br class="m_-8284271755943322836gmail_msg"></li><li \
class="m_-8284271755943322836gmail_msg">Sent SIGSTOP to process tree: sh, command<br \
class="m_-8284271755943322836gmail_msg"></li><li \
class="m_-8284271755943322836gmail_msg">Sent SIGTERM to process tree: sh, command<br \
class="m_-8284271755943322836gmail_msg"></li><li \
class="m_-8284271755943322836gmail_msg">Wait for processes to finish or grace period \
to elapse<br class="m_-8284271755943322836gmail_msg"></li><li \
class="m_-8284271755943322836gmail_msg">sh finish while command could be still \
running and attached to init<br class="m_-8284271755943322836gmail_msg"></li><li \
class="m_-8284271755943322836gmail_msg">Sent SIGKILL to process tree: command<br \
class="m_-8284271755943322836gmail_msg"></li></ol><div \
class="m_-8284271755943322836gmail_msg">I notice that SIGKILL is not sent and \
executor finished when sh returns. When Mesos is running with POSIX contenerizer this \
leads command to live forever (if it ignores SIGTERM). When contenerizer is used \
command is killed when it's container is destroyed.</div><div \
class="m_-8284271755943322836gmail_msg"><br \
class="m_-8284271755943322836gmail_msg"></div><div \
class="m_-8284271755943322836gmail_msg">Is this desired behavior? How to use grace \
period with default executor?</div><div class="m_-8284271755943322836gmail_msg"><br \
class="m_-8284271755943322836gmail_msg"></div><div \
class="m_-8284271755943322836gmail_msg">Thanks</div><span \
class="m_-8284271755943322836m_7808437058368462897m_1115538748178419118HOEnZb \
m_-8284271755943322836gmail_msg"><font color="#888888" \
class="m_-8284271755943322836gmail_msg"><div \
class="m_-8284271755943322836gmail_msg">Tomek</div></font></span></div> \
</blockquote></div><br class="m_-8284271755943322836gmail_msg"><br clear="all" \
class="m_-8284271755943322836gmail_msg"><div \
class="m_-8284271755943322836gmail_msg"><br \
class="m_-8284271755943322836gmail_msg"></div></div><div class="gmail_extra \
m_-8284271755943322836gmail_msg">-- <br class="m_-8284271755943322836gmail_msg"><div \
class="m_-8284271755943322836m_7808437058368462897m_1115538748178419118gmail_signature \
m_-8284271755943322836gmail_msg" data-smartmail="gmail_signature">Best Regards,<br \
class="m_-8284271755943322836gmail_msg"><div \
class="m_-8284271755943322836gmail_msg">Haosdent Huang</div></div> \
</div></blockquote></div></blockquote></div> \
</div></div></blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic