[prev in list] [next in list] [prev in thread] [next in thread]
List: apache-httpd-users
Subject: [users@httpd] Re: [EXT] [users@httpd] Apache server using SIGTERM then SIGKILL on stop/restart
From: "Saux, Stephane" <ssaux () sfchronicle ! com ! INVALID>
Date: 2023-01-13 17:28:32
Message-ID: 6E594285-4019-4392-8A5C-2EEE14E9C653 () sfchronicle ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
FWIW. It's not Apache related. It's systemd that kills the process using cgroup
https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
From: "Saux, Stephane" <ssaux@sfchronicle.com.INVALID>
Reply-To: "users@httpd.apache.org" <users@httpd.apache.org>
Date: Friday, January 13, 2023 at 7:47 AM
To: "users@httpd.apache.org" <users@httpd.apache.org>
Subject: [EXT] [users@httpd] Apache server using SIGTERM then SIGKILL on stop/restart
I have a system that uses a web server (Apache) to allow non-technical users to start \
long-running jobs on our systems using a browser. The system starts the job and \
returns info that allows the browser to poll a JSON file that the long-running job \
maintains to give the user status info about the job (simplified description.)
But I determined that if Apache restarts on that server, it kills that process. I had \
done this about 10 years ago and it worked without doing this.
I tried to detach the process by forking, and I also tried to have catch SIGTERM. As \
a result, Apache restart takes over a minute because something in the server (or \
apachctl I'm not sure) tries a number of time to send that process a SIGTERM (which \
it ignores) and then just sends it a SIGKILL.
Here's what syslog shows:
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: State 'stop-final-sigterm' timed \
out. Killing.
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Killing process 113975 (php) with \
signal SIGKILL.
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Failed with result 'timeout'.
Jan 13 04:09:57 wcmdev systemd[1]: Stopped The Apache HTTP Server.
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Found left-over process 113975 \
(php) in control group while starting unit. Ignoring.
Jan 13 04:09:57 wcmdev systemd[1]: This usually indicates unclean termination of a \
previous run, or service implementation deficiencies.
Jan 13 04:09:57 wcmdev systemd[1]: Starting The Apache HTTP Server...
Here's what the long running process looks like. Note that the PPID is 1, so it is \
detached.
root@wcmdev:/var/www/wcm# ps -f -p 110733
UID PID PPID C STIME TTY TIME CMD
www-data 110733 1 0 02:58 ? 00:00:00 php \
/media/sf_Shared/wcm/src/business/api/system/../../../jobs/daemon/testDaemon.php \
--id=3852114
I search the list for stop-final-sigterm, and could not find anything.
root@wcmdev:/var/www/wcm# /usr/sbin/apache2 -v
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2020-08-12T19:46:17
root@wcmdev:/var/www/wcm# uname -a
Linux wcmdev 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 \
x86_64 x86_64 GNU/Linux
[Attachment #3 (text/html)]
<html xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Calibri Light";
panose-1:2 15 3 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">FWIW. It's not Apache related. \
It's systemd that kills the process using cgroup<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"><a \
href="https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt">https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span \
style="color:black">"Saux, Stephane" \
<ssaux@sfchronicle.com.INVALID><br> <b>Reply-To: \
</b>"users@httpd.apache.org" <users@httpd.apache.org><br> <b>Date: \
</b>Friday, January 13, 2023 at 7:47 AM<br> <b>To: \
</b>"users@httpd.apache.org" <users@httpd.apache.org><br> <b>Subject: \
</b>[EXT] [users@httpd] Apache server using SIGTERM then SIGKILL on \
stop/restart<o:p></o:p></span></p> </div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">I have a system that uses a web \
server (Apache) to allow non-technical users to start long-running jobs on our \
systems using a browser. The system starts the job and returns info that allows the \
browser to poll a JSON file that the long-running job maintains to give the user \
status info about the job (simplified description.)<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt">But I determined that if Apache \
restarts on that server, it kills that process. I had done this about 10 years ago \
and it worked without doing this.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">I tried to detach the process by forking, and I also tried \
to have catch SIGTERM. As a result, Apache restart takes over a minute because \
something in the server (or apachctl I'm not sure) tries a number of time to send \
that process a SIGTERM (which it ignores) and then just sends it a \
SIGKILL.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">Here's what syslog shows:<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier \
New"">Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: State \
'stop-final-sigterm' timed out. Killing.<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier \
New"">Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Killing process 113975 \
(php) with signal SIGKILL.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Courier New"">Jan 13 04:09:57 wcmdev \
systemd[1]: apache2.service: Failed with result 'timeout'.<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier \
New"">Jan 13 04:09:57 wcmdev systemd[1]: Stopped The Apache HTTP \
Server.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Courier New"">Jan 13 04:09:57 wcmdev \
systemd[1]: apache2.service: Found left-over process 113975 (php) in control group \
while starting unit. Ignoring.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Courier New"">Jan 13 04:09:57 wcmdev \
systemd[1]: This usually indicates unclean termination of a previous run, or service \
implementation deficiencies.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Courier New"">Jan 13 04:09:57 wcmdev \
systemd[1]: Starting The Apache HTTP Server...<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier \
New""><o:p> </o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri Light",sans-serif">Here's what \
the long running process looks like. Note that the PPID is 1, so it is \
detached.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri \
Light",sans-serif"><o:p> </o:p></span></p> <p class="p1"><span \
class="s1">root@wcmdev:/var/www/wcm# ps -f -p 110733</span><o:p></o:p></p> <p \
class="p1"><span class="s1">UID</span><span class="apple-converted-space"> \
</span><span class="s1">PID</span><span \
class="apple-converted-space"> </span> <span class="s1">PPID</span><span \
class="apple-converted-space"> </span><span class="s1">C STIME TTY</span><span \
class="apple-converted-space"> </span><span \
class="s1">TIME CMD</span><o:p></o:p></p> <p class="p1"><span \
class="s1">www-data</span><span class="apple-converted-space"> </span><span \
class="s1">110733 </span><span class="apple-converted-space"> \
</span> <span class="s1">1</span><span class="apple-converted-space"> \
</span><span class="s1">0 02:58 ?</span><span class="apple-converted-space"> \
</span><span class="s1">00:00:00 php \
/media/sf_Shared/wcm/src/business/api/system/../../../jobs/daemon/testDaemon.php \
--id=3852114</span><o:p></o:p></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Courier \
New""><o:p> </o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri \
Light",sans-serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri Light",sans-serif">I search \
the list for stop-final-sigterm, and could not find anything.<o:p></o:p></span></p> \
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri \
Light",sans-serif"><o:p> </o:p></span></p> <p class="p1"><span \
class="s1">root@wcmdev:/var/www/wcm# /usr/sbin/apache2 -v</span><o:p></o:p></p> <p \
class="p1"><span class="s1">Server version: Apache/2.4.41 \
(Ubuntu)</span><o:p></o:p></p> <p class="p1"><span class="s1">Server built: \
</span><span class="apple-converted-space"> </span><span \
class="s1">2020-08-12T19:46:17</span><o:p></o:p></p> <p class="p1"><span \
class="s1">root@wcmdev:/var/www/wcm# uname -a</span><o:p></o:p></p> <p \
class="p1"><span class="s1">Linux wcmdev 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 \
10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux</span><o:p></o:p></p> <p \
class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier \
New""><o:p> </o:p></span></p> </div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic