[prev in list] [next in list] [prev in thread] [next in thread] 

List:       apache-httpd-bugs
Subject:    [Bug 62009] New: scoreboard is full with worker MPM
From:       bugzilla () apache ! org
Date:       2018-01-17 13:57:54
Message-ID: bug-62009-7868 () https ! bz ! apache ! org/bugzilla/
[Download RAW message or body]

https://bz.apache.org/bugzilla/show_bug.cgi?id=62009

            Bug ID: 62009
           Summary: scoreboard is full with worker MPM
           Product: Apache httpd-2
           Version: 2.4.29
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_http2
          Assignee: bugs@httpd.apache.org
          Reporter: a.abfalterer@gmail.com
  Target Milestone: ---

I am currently investigating a problem (Apache/2.4.29 with mod_http2 v1.10.12)
related to worker MPM that leads to excessive termination and starting of
processes, sometimes also up to the termination of all httpd processes, thus to
the outage of the entire proxy. An outage occurs in connection with the log
entry "scoreboard is full, not at MaxRequestWorkers".

It looks as if the behavior is caused by mod_http2.

# Reproduction

The behavior can be aggravated if few processes are started, I use following
settings

ServerLimit 4
StartServers 1
MaxRequestWorkers 12
MinSpareThreads 3
MaxSpareThreads 6
ThreadsPerChild 3

After a restart, a large number of simultaneous requests (I generate 100
simultaneous requests) must be made. This is done so often (mostly 2-4 times
with a pause in the middle of each bulk) until httpd has 4 child processes.
Then you stop so that httpd becomes idle. Let's name this state "idle-4-proc". 

The misbehavior is now manifested by the fact that excess processes are not
killed. Actually, there should only be a maximum of 2 processes
(MaxSpareThreads 6).

If in this state another bulk of requests is sent off, httpd begins to stop and
start processes like crazy. Now, sending some other bulks sometimes (not
always) lead to the already mentioned "scoreboard is full" error.

# Analysis

I could find out following

## httpd without mod_http2

If httpd is built without mod_http2, the main process recognizes in idle-4-proc
state that too many processes are available and signals by means of
AP_MPM_PODX_GRACEFUL that a process must be terminated. One of the child
processes reads AP_MPM_PODX_GRACEFUL from the POD, sends the signal SIGHUP,
which is intercepted by the listener thread. The listener thread then sends
SIGTERM to its PID to terminate all worker threads. In this way, the surplus
processes are terminated cleanly.

## httpd with mod_http2

However, if httpd is build with mod_http2 the listener thread does not receive
SIGHUP and therefore cannot send SIGTERM to the worker threads. As a result,
the child process is not terminated, httpd continues to send
AP_MPM_PODX_GRACEFUL and all child processes end up in an inconsistent state
(marked as terminated, but not really terminated). When requests now arrive,
all listener threads (of semi-terminated child processes) recognize that they
have to terminate and so they finally send SIGTERM to its PID. This causes the
misbehavior of httpd, which now starts too many new child processes.

The problem that listener threads does not receive SIGHUP seems to be that the
signal is caught by an h2-worker. This can be confirmed by attaching to an
child-process before idle-4-proc state and then sending off the bulks: one can
see that it is an h2-worker thread (instead of the listener thread) that
"handles" SIGHUP in worker.c:dummy_signal_handler().

# Workaround

The problem can be workaround-ed by blocking SIGHUP for each h2-worker, for
example by adding

    sigset_t sig_mask;

    sigemptyset(&sig_mask);
    sigaddset(&sig_mask, SIGHUP);
#if defined(SIGPROCMASK_SETS_THREAD_MASK)
    sigprocmask(SIG_BLOCK, &sig_mask, NULL);
#else
    pthread_sigmask(SIG_BLOCK, &sig_mask, NULL);
#endif


in h2_workers.c:slot_run(). Doing so, I can observe same behavior as like with
httpd without mod_http2.

I'd appreciate it if someone could confirm the misbehavior and my analysis.

Thanks, Armin

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

[prev in list] [next in list] [prev in thread] [next in thread]