[prev in list] [next in list] [prev in thread] [next in thread] 

List:       apache-modperl
Subject:    Re: Cache refresh each 50 queries?
From:       James Smith <js5 () sanger ! ac ! uk>
Date:       2016-10-05 11:12:40
Message-ID: cd8d47f4-6f0c-54a7-27bc-dd0dac9d1f86 () sanger ! ac ! uk
[Download RAW message or body]

You can look at Apache::SizeLimit as an alternative - this is designed 
to cope with applications which occassionaly "leak memory"..

If one requests uses a lot of memory it will not be recovered -- Perl 
doesn't hand this memory back - so subsequent requests are handled by 
the inflated process.

We use mod_perl handlers to track this to see which requests cause the 
memory inflation to happen - these can then be looked at to see if they 
can be optimized! Check that large datastructures are not being copied 
around - avoiding processing one large data structure into another large 
data structure etc....

On 10/5/2016 11:39 AM, A. Warnier wrote:
> Hi again.
>
> I want to correct somewhat what I was saying below, in "Additional hint".
> See : 
> http://perl.apache.org/docs/1.0/guide/performance.html#Sharing_Memory
>
> What I was saying about "leaking memory" below remains true.
>
> But even with applications which do not "leak memory" per se, there 
> will anyway be some increase in the amount of memory that each Apache 
> child is using, over time.
> So the "best" setting for "MaxRequestsPerChild", in an Apache/mod_perl 
> scenario, may not be 0 ( = unlimited).  Instead, use some relatively 
> large number, such as 1000 for example.
> That will cause a child to serve 1000 requests before it gets replaced 
> by a new one.
> And having a slightly longer delay every 1000 requests, would probably 
> not even be noticed. (Of course, if your server crashes with an OOM 
> before you reach 1000, then you will have to reduce that number, 
> experimentally).
>
> The reason is well explained in that page above.
>
>
> On 05.10.2016 11:30, A. Warnier wrote:
>> Additional hint :
>>
>> Normally, when Apache goes through the "effort" of setting up a new 
>> child process, you
>> would want that this child process then runs for as long as possible 
>> (for as many requests
>> as possible), to avoid the repeated overhead of restarting another 
>> child process.
>> In an Apache prefork configuration, you would do this by setting
>>
>> MaxRequestsPerChild 0   (or a very high number)
>>
>> (as per 
>> http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild)
>>
>> However, some perl modules/applications are known to "leak memory" 
>> (meaning that each time
>> they are called, the Apache child memory footprint increases a bit, 
>> and never goes down
>> anymore).  And after a while, depending on how busy the site is and 
>> how many times that
>> application is called, Apache may crash if it does not have enough 
>> memory anymore.
>>
>> That is why, in some cases, you would set this MaxRequestsPerChild to 
>> a value different
>> from 0, to limit the number of requests that any Apache child would 
>> process, before it is
>> forcefully terminated by Apache, and replaced by a new one.
>> (The exact number used, would depend on how much additional memory it 
>> uses after each
>> request).
>> Terminating this child process would return the used memory to the 
>> OS, and the new child
>> would start again fresh, with a minimum amount of memory.
>> Then it will start growing again, until it hits the limit again, etc..
>>
>> So, setting MaxRequestsPerChild <> 0, is in fact a "quick-and-dirty 
>> fix" for a problem of
>> the application (it should not do that, leaking memory). And it would 
>> be better to fix the
>> application, so that it can run without leaking memory..
>>
>> P.S.
>> You still have not indicated what your OS is, or what kind of Apache 
>> MPM you are using.
>> If you need more help, please do that the next time you post a 
>> message here.
>>
>>
>> On 05.10.2016 10:53, SUZUKI Arthur wrote:
>>> Hello David, André,
>>>
>>> the server we're working with is fully dedicated to Koha.
>>>
>>> Thank you André for your hints about Apache configurations.
>>>
>>> @David, same thing, I'll try to preload C4::ILSDI:Services and see 
>>> if it helps.
>>>
>>> Thanks to both of you, I'll let you know about the results.
>>> Cheers,
>>> Arthur
>>>
>>> Le 05/10/2016 à 01:20, David Cook a écrit :
>>>> Sounds like André is on the right track. I've certainly run into a 
>>>> similar
>>>> issue (with a non-Koha app).
>>>>
>>>> However, because I was using the Catalyst framework, I was able to 
>>>> just
>>>> preload the entire app, so that the Perl modules were loaded into 
>>>> the Apache
>>>> master process before forking, and that did the trick. That app is 
>>>> a lot
>>>> smaller than Koha though too. This case is a bit more complicated 
>>>> since Koha
>>>> isn't really a MVC app, but you could look at the Koha Plack 
>>>> examples and
>>>> see which modules they pre-load.
>>>>
>>>> You might also want to run ilsdi.pl with Devel::NYTProf to identify 
>>>> what
>>>> exactly is slowing down that 1s long query.
>>>>
>>>> I've only played with PerlOptions +Parent a bit, but I'm guessing 
>>>> that you
>>>> have multiple mod_perl apps, and that we're not actually seeing 
>>>> your entire
>>>> httpd.conf configuration relating to Koha, right?
>>>>
>>>> David Cook
>>>> Systems Librarian
>>>> Prosentient Systems
>>>> 72/330 Wattle St
>>>> Ultimo, NSW 2007
>>>> Australia
>>>>
>>>> Office: 02 9212 0899
>>>> Direct: 02 8005 0595
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: André Warnier [mailto:aw@ice-sa.com]
>>>>> Sent: Tuesday, 4 October 2016 10:02 PM
>>>>> To: modperl@perl.apache.org
>>>>> Subject: Re: Cache refresh each 50 queries?
>>>>>
>>>>> Hi.
>>>>>
>>>>> On 04.10.2016 10:17, SUZUKI Arthur wrote:
>>>>> [...]
>>>>>
>>>>>> On 03.10.2016 16:57, SUZUKI Arthur wrote:
>>>>>>>> Hello List,
>>>>>>>> I'm sending this message to know if there are some hints/tips to
>>>>>>>> help with the problem we're facing.
>>>>>>>> The problem is that for a same query repeated over time, reply 
>>>>>>>> time
>>>>>>>> can be as short as 5ms and as long as 1s.
>>>>>>>>
>>>>>>>> Since there is neither correlation with CPU load or RAM usage, nor
>>>>>>>> any networking constraints, we think it is due to the cache 
>>>>>>>> refresh
>>>>>>>> mechanism.
>>>>>>>>
>>>>>>>> The problem occurs at random times, with a probability of around
>>>>>>>> 1/50 queries (empirical data).
>>>>>>>>
>>>>>>>> Is there any configuration option which could help?
>>>>>>>>
>>>>>>>> Our current httpd.conf contains the following:
>>>>>>>>
>>>>>>>> PerlModule ModPerl::Registry
>>>>>>>> PerlOptions +Parent
>>>>>>>> PerlSwitches -I/home/koha/src
>>>>>>>>
>>>>>>>> <Files "ilsdi.pl">
>>>>>>>>     SetHandler perl-script
>>>>>>>>     # more faster, link with worker
>>>>>>>>     PerlResponseHandler ModPerl::Registry
>>>>>>>>     # less faster, link with prefork
>>>>>>>>     #PerlResponseHandler ModPerl::PerlRun
>>>>>>>>     Options +ExecCGI
>>>>>>>>     PerlOptions +ParseHeaders
>>>>>>>> </Files>
>>>>>>>>
>>>>>>>> Thanks a lot in advance for your replies.
>>>>>>>> Best regards,
>>>>> You still do not provide enough information about your server
>>>> configuration
>>>>> (the part not mod_perl specific) to really make a narrow guess, 
>>>>> but let me
>>>> try
>>>>> anyway :
>>>>>
>>>>> One thing that might explain a seemingly-long processing time for a
>>>> request
>>>>> approximately once in 50 calls, is if there was something special and
>>>>> resource-intensive which happens once per approximately 50 calls, 
>>>>> right ?
>>>>>
>>>>> One thing which /might/ explain this, /perhaps/ and depending on your
>>>>> configuration, is if approximately once in 50 calls, Apache had to 
>>>>> fork a
>>>> new
>>>>> child process, to handle the call. This new child process would 
>>>>> then start
>>>> with
>>>>> a brand-new, fresh Perl interpreter, which might need to recompile 
>>>>> some
>>>>> modules before it even starts handling the request.
>>>>>
>>>>> Look for example here :
>>>>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
>>>>> and your configuration line above :
>>>>>    PerlOptions +Parent
>>>>>
>>>>> In that configuration, the cgi-bin script which you are running 
>>>>> (and all
>>>> its
>>>>> dependencies), is not compiled by the embedded perl until it has 
>>>>> been run
>>>> at
>>>>> least once.
>>>>> So the first execution will take more time than the following ones.
>>>>> Now if the rest of your configuration is so that every 50 requests 
>>>>> or so,
>>>>> Apache starts a new child with a new perl interpreter, that may 
>>>>> explain
>>>> the
>>>>> symptoms that you are observing.
>>>>>
>>>>> When Apache handles a request (if it is a "prefork" MPM), it will 
>>>>> look for
>>>> a
>>>>> child Apache which is free, to pass the request for execution.  If 
>>>>> the
>>>> server is
>>>>> lightly loaded, it may be that the child to which it passes the next
>>>> request is
>>>>> always the same, because it is always done with the previous 
>>>>> request, and
>>>>> free, when the next request comes in.
>>>>> But if this child process somehow has a limit to how many requests 
>>>>> it may
>>>>> process before it dies, and this limit was around 50, then somewhere
>>>> around
>>>>> the 50th request, another child would get the next request, and 
>>>>> for this
>>>> child
>>>>> it would be the first one (or the first one with /this/ cgi-bin 
>>>>> script).
>>>>>
>>>>> There are roughly similar phenomenons that might happen in other 
>>>>> kinds of
>>>>> Apache MPM's, which is why it is important to know which one you are
>>>> using,
>>>>> and with which setup parameters.
>>>>>
>>>>> And yes, there are ways to improve this.  But again, we would need 
>>>>> to know
>>>>> more about your configuration in order to make suggestions.
>>>>
>>>>
>>>
>>
>



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
[prev in list] [next in list] [prev in thread] [next in thread]