[prev in list] [next in list] [prev in thread] [next in thread]
List: apache-modperl
Subject: Re:
From: Adam Prime <adam.prime () utoronto ! ca>
Date: 2021-02-07 13:45:08
Message-ID: CBC3BF5B-5DD5-497D-BB67-1AE4391806E3 () utoronto ! ca
[Download RAW message or body]
There is one other thing you can do relatively easily that may get you a marginal \
gain when Apache spins up new children. Load some or all of your Perl dependencies \
before Apache forks. There is also an opportunity to load other static resources at \
this time, but that can get a little more involved and/or confusing. There is some \
documentation here:
https://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File
Adam
> On Feb 7, 2021, at 8:15 AM, Steven Haigh <netwiz@crc.id.au> wrote:
>
>
> In fact, I just realised that 'ab' test is rather restrictive.... So here's a bit \
> more of an extended test:
> # ab -k -n 1000 -c 32
>
> Apache + ExecCGI:
> Requests per second: 14.26 [#/sec] (mean)
> Time per request: 2244.181 [ms] (mean)
> Time per request: 70.131 [ms] (mean, across all concurrent requests)
>
> Apache + mod_perl (ModPerl::PerlRegistry):
> Requests per second: 132.14 [#/sec] (mean)
> Time per request: 242.175 [ms] (mean)
> Time per request: 7.568 [ms] (mean, across all concurrent requests)
>
> Interestingly, without Keepalives, the story is much the same:
>
> # ab -n 1000 -c 32
>
> Apache + ExecCGI:
> Requests per second: 14.15 [#/sec] (mean)
> Time per request: 2260.875 [ms] (mean)
> Time per request: 70.652 [ms] (mean, across all concurrent requests)
>
> Apache + mod_perl (ModPerl::PerlRegistry):
> Requests per second: 154.48 [#/sec] (mean)
> Time per request: 207.140 [ms] (mean)
> Time per request: 6.473 [ms] (mean, across all concurrent requests)
>
> Running some benchmarks across various parts of my site made me realise I also had \
> some RewriteRules in the apache config that still had H=cgi-script - changed those \
> to H=perl-script and saw similar improvements:
> ExecCGI - Requests per second: 11.84 [#/sec] (mean)
> mod_perl - Requests per second: 130.97 [#/sec] (mean)
>
> That's quite some gains for a days work.
>
> --
> Steven Haigh
>
> 📧 netwiz@crc.id.au
> 💻 https://www.crc.id.au
>
> > On Sun, Feb 7, 2021 at 23:58, Steven Haigh <netwiz@crc.id.au> wrote:
> > Interestingly, I did get things working with ModPerl::PerlRegistry.
> >
> > What I couldn't find *anywhere* is that the data I was loading in Template \
> > Toolkit was included in the file in the __DATA__ area - which causes mod_perl to \
> > fall over!
> > The only way I managed to find this was the following error in the *system* \
> > /var/log/httpd/error_log (didn't show up in the vhost error_log!): readline() on \
> > unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm \
> > line 638.
> > Took me a LONG time to find a vague post that reading in lines from <DATA> kills \
> > mod_perl. Not sure why - but I stripped all the templates out and put them in a \
> > file instead and re-wrote that bit of code, and things started working.
> > I had to fix a few lib path issues, but after getting my head around that, most \
> > things seem to work as before - however I don't notice much of an improvement in \
> > execution times, I do see this improvement using 'ab -n 100 -c32':
> > Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
> > Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
> >
> > This is obviously a good thing.
> >
> > I haven't gotten into the preload or DBI sharing yet - as that'll end up needing \
> > a bit of a rewrite of code to take advantage of. I'd be open to suggestions here \
> > from those who have done it in the past to save me going down some dead ends :D
> > --
> > Steven Haigh
> >
> > 📧 netwiz@crc.id.au
> > 💻 https://www.crc.id.au
> >
> > > On Sun, Feb 7, 2021 at 12:49, James Smith <js5@sanger.acuk> wrote:
> > > As welsey said – try Registry, that was the standard way of using mod_perl to \
> > > cache perl in the server – but your problem might be due to the note in \
> > > PerlRun…
> > > https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description
> > > META: document that for now we don't chdir() into the script's dir, because it \
> > > affects the whole process under threads. ModPerl::PerlRunPrefork should be used \
> > > by those who run only under prefork MPM. {tbh most people don't use mod perl \
> > > under threads anyway as there isn't really a gain from using them}
> > > It suggests you use ModPerl/PerlRunPrefork – as this does an additional step \
> > > to cd to the script directory – which might be your issue….
> > > From: Steven Haigh <netwiz@crc.id.au>
> > > Sent: 07 February 2021 01:00
> > > To: modperl@perl.apache.org
> > > Subject: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
> > >
> > > Hi all,
> > >
> > > So for many years I've been slack and writing perl scripts to do various things \
> > > - but never needed more than the normal apache +ExecCGI and Template Toolkit.
> > > One of my sites has become a bit more popular, so I'd like to spend a bit of \
> > > time on performance. Currently, I'm seeing ~300-400ms of what I believe to be \
> > > execution time of the script loading, running, and then blatting its output to \
> > > STDOUT and the browser can go do its thing.
> > > I believe most of the delay would be to do with loading perl, its modules etc \
> > > etc
> > > I know that the current trend would be to re-write the entire site in a more \
> > > modern, daemon based solution - and I started down the Mojolicious path - but \
> > > the amount of re-writing to save 1/3rd of a second seems to be excessive
> > > Would I be correct in thinking that mod_perl would help in this case?
> > >
> > > I did try a basic test, but I have a 'use functions' in all my scripts that \
> > > loads a .pm with some global vars and a lot of common subs - and for whatever \
> > > reason (can't find anything on Google as to why), none of the subs are \
> > > recognised in the main script when loaded via ModPerl::PerlRun.
> > > So throwing it out to the list - am I on the right track? wasting my time? or \
> > > just a simple mistake?
> > > --
> > > Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au [crc.id.au]
> > > -- The Wellcome Sanger Institute is operated by Genome Research Limited, a \
> > > charity registered in England with number 1021457 and a company registered in \
> > > England with number 2742969, whose registered office is 215 Euston Road, \
> > > London, NW1 2BE.
[Attachment #3 (text/html)]
<html><head><meta http-equiv="content-type" content="text/html; \
charset=utf-8"></head><body dir="auto"><div dir="ltr">There is one other thing you \
can do relatively easily that may get you a marginal gain when Apache spins up new \
children. Load some or all of your Perl dependencies before Apache forks. There is \
also an opportunity to load other static resources at this time, but that can get a \
little more involved and/or confusing. There is some documentation \
here: </div><div dir="ltr"><br></div><div dir="ltr"><a \
href="https://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File">https://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File</a></div><div \
dir="ltr"><br></div><div dir="ltr">Adam</div><div dir="ltr"><br></div><div \
dir="ltr"><br><blockquote type="cite">On Feb 7, 2021, at 8:15 AM, Steven Haigh \
<netwiz@crc.id.au> wrote:<br><br></blockquote></div><blockquote \
type="cite"><div dir="ltr"><div id="geary-body" dir="auto"><div>In fact, I just \
realised that 'ab' test is rather restrictive.... So here's a bit more of an extended \
test:</div><div><br></div><div># ab -k -n 1000 -c 32</div><div><br></div><div>Apache \
+ ExecCGI:</div><div><span style="white-space: pre-wrap;"> </span>Requests per \
second: 14.26 [#/sec] (mean)</div><div><div><div><div><span \
style="white-space: pre-wrap;"> </span>Time per request: \
2244.181 [ms] (mean)</div><div><span style="white-space: pre-wrap;"> </span>Time per \
request: 70.131 [ms] (mean, across all concurrent \
requests)</div></div></div></div><div><br></div><div>Apache + mod_perl \
(ModPerl::PerlRegistry): </div><div><span style="white-space: \
pre-wrap"> </span><span style="white-space: pre-wrap;">Requests per second: 132.14 \
[#/sec] (mean)</span></div><div><div><div><span style="white-space: \
pre-wrap;"> </span>Time per request: 242.175 [ms] \
(mean)</div><div><span style="white-space: pre-wrap;"> </span>Time per request: \
7.568 [ms] (mean, across all concurrent \
requests)</div></div><div><br></div></div><div>Interestingly, without Keepalives, the \
story is much the same:</div><div><br></div><div># ab -n 1000 -c \
32</div><div><br></div><div>Apache + ExecCGI:</div><div><div><span \
style="white-space: pre-wrap;"> </span>Requests per second: 14.15 \
[#/sec] (mean)</div><div><span style="white-space: pre-wrap;"> </span>Time per \
request: 2260.875 [ms] (mean)</div><div><span \
style="white-space: pre-wrap;"> </span>Time per request: 70.652 \
[ms] (mean, across all concurrent \
requests)</div></div><div><br></div><div><div>Apache + mod_perl \
(ModPerl::PerlRegistry): </div><div><span style="white-space: \
pre-wrap;"> </span>Requests per second: 154.48 [#/sec] \
(mean)</div><div><span style="white-space: pre-wrap;"> </span>Time per request: \
207.140 [ms] (mean)</div><div><span style="white-space: \
pre-wrap;"> </span>Time per request: 6.473 [ms] (mean, across \
all concurrent requests)</div></div><div><br></div><div>Running some benchmarks \
across various parts of my site made me realise I also had some RewriteRules in the \
apache config that still had H=cgi-script - changed those to H=perl-script and saw \
similar improvements:</div><div><div><br></div><div><span style="white-space: \
pre-wrap;"> ExecCGI - </span>Requests per second: 11.84 [#/sec] \
(mean)</div></div><div><span style="white-space: pre-wrap;"> mod_perl - \
</span>Requests per second: 130.97 [#/sec] \
(mean)</div><div><br></div><div>That's quite some gains for a days \
work.</div></div><div id="geary-signature" dir="auto"><div><br></div><span \
style="white-space: pre-wrap; background-color: rgb(255, 255, \
255);">--</span><div><span style="white-space: pre-wrap; background-color: rgb(255, \
255, 255);">Steven Haigh
📧 </span><a href="mailto:netwiz@crc.id.au" style="white-space: \
pre-wrap;">netwiz@crc.id.au</a><span style="white-space: pre-wrap; background-color: \
rgb(255, 255, 255);"> 💻 </span><a href="https://www.crc.id.au/" \
style="white-space: pre-wrap;">https://www.crc.id.au</a></div></div><div \
id="geary-quote" dir="auto"><br>On Sun, Feb 7, 2021 at 23:58, Steven Haigh \
<netwiz@crc.id.au> wrote:<br><blockquote type="cite"><div id="geary-body" \
dir="auto"><div>Interestingly, I did get things working with \
ModPerl::PerlRegistry.</div><div><br></div><div>What I couldn't find *anywhere* is \
that the data I was loading in Template Toolkit was included in the file in the \
__DATA__ area - which causes mod_perl to fall over!</div><div><br></div><div>The only \
way I managed to find this was the following error in the *system* \
/var/log/httpd/error_log (didn't show up in the vhost \
error_log!):</div><div><div><span style="white-space: pre-wrap;"> </span>readline() \
on unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm line \
638.</div></div><div><br></div><div>Took me a LONG time to find a vague post that \
reading in lines from <DATA> kills mod_perl. Not sure why - but I stripped all \
the templates out and put them in a file instead and re-wrote that bit of code, and \
things started working.</div><div><br></div><div>I had to fix a few lib path issues, \
but after getting my head around that, most things seem to work as before - however I \
don't notice much of an improvement in execution times, I do see this improvement \
using 'ab -n 100 -c32':</div><div><br></div><div><div><span style="white-space: \
pre-wrap;"> </span>Apache + ExecCGI: Requests per second: 13.50 [#/sec] \
(mean)</div><div><span style="white-space: pre-wrap;"> </span>Apache + mod_perl: \
Requests per second: 59.81 [#/sec] \
(mean)</div></div><div><br></div><div>This is obviously a good \
thing.</div><div><br></div><div>I haven't gotten into the preload or DBI sharing yet \
- as that'll end up needing a bit of a rewrite of code to take advantage of. I'd be \
open to suggestions here from those who have done it in the past to save me going \
down some dead ends :D</div></div><div id="geary-signature" dir="auto"><div><span \
style="white-space: pre-wrap; background-color: rgb(255, 255, \
255);"><br></span></div><span style="white-space: pre-wrap; background-color: \
rgb(255, 255, 255);">--</span><div><span style="white-space: pre-wrap; \
background-color: rgb(255, 255, 255);">Steven Haigh
📧 </span><a href="mailto:netwiz@crc.id.au" style="white-space: \
pre-wrap;">netwiz@crc.id.au</a><span style="white-space: pre-wrap; background-color: \
rgb(255, 255, 255);"> 💻 </span><a href="https://www.crc.id.au/" \
style="white-space: pre-wrap;">https://www.crc.id.au</a></div></div><div \
id="geary-quote" dir="auto"><br>On Sun, Feb 7, 2021 at 12:49, James Smith \
<js5@sanger.acuk> wrote:<br><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">As welsey said – try \
Registry, that was the standard way of using mod_perl to cache perl in the server \
– but your problem might be due to the note in PerlRun…<br> <br>
<a href="https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description">https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description</a><br>
</span><span style="font-size:12.0pt;color:black;background:white">META: document \
that for now we don't chdir() into the script's dir, because it affects the whole \
process under threads. <code><span \
style="font-family:"Calibri",sans-serif"><a \
href="https://perl.apache.org/docs/20/api/ModPerl/PerlRunPrefork.html">ModPerl::PerlRunPrefork</a></span></code> should
be used by those who run only under prefork MPM.</span><span \
style="mso-fareast-language:EN-US"><br> {tbh most people don't use mod perl under \
threads anyway as there isn't really a gain from using them}<br> <br>
It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd \
to the script directory – which might be your issue….<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div \
style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p \
class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Steven \
Haigh <netwiz@crc.id.au> <br>
<b>Sent:</b> 07 February 2021 01:00<br>
<b>To:</b> modperl@perl.apache.org<br>
<b>Subject:</b> Moving ExecCGI to mod_perl - performance and custom 'modules' \
[EXT]<o:p></o:p></span></p> </div>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="geary-body">
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So for many years I've been slack and writing perl scripts to do \
various things - but never needed more than the normal apache +ExecCGI and Template \
Toolkit.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One of my sites has become a bit more popular, so I'd like to \
spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I \
believe to be execution time of the script loading, running, and then blatting its \
output to STDOUT and the browser can go do its thing. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I believe most of the delay would be to do with loading perl, \
its modules etc etc<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I know that the current trend would be to re-write the entire \
site in a more modern, daemon based solution - and I started down the Mojolicious \
path - but the amount of re-writing to save 1/3rd of a second seems to be \
excessive<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Would I be correct in thinking that mod_perl would help in this \
case?<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I did try a basic test, but I have a 'use functions' in all my \
scripts that loads a .pm with some global vars and a lot of common subs - and for \
whatever reason (can't find anything on Google as to why), none of the subs are \
recognised in the main script when loaded via ModPerl::PerlRun.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So throwing it out to the list - am I on the right track? \
wasting my time? or just a simple mistake?<o:p></o:p></p> </div>
</div>
<div id="geary-signature">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span \
style="color:black;background:white">--</span><o:p></o:p></p> <div>
<p class="MsoNormal"><span style="color:black;background:white">Steven Haigh </span>
<span style="font-family:"Segoe UI \
Emoji",sans-serif;color:black;background:white">📧</span><span \
style="color:black;background:white"> </span><a \
href="mailto:netwiz@crc.id.au">netwiz@crc.id.au</a><span \
style="color:black;background:white"> </span><span style="font-family:"Segoe UI \
Emoji",sans-serif;color:black;background:white">💻</span><span \
style="color:black;background:white"> </span><a \
href="www.crc.id.au_&d=DwMFaQ" rel="nofollow">https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ> \
&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m= \
bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=">https://www.crc.id.au
[crc.id.au]</a><o:p></o:p></p>
</div>
</div>
</div>
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
</blockquote></div></blockquote></div></div></blockquote></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic