[prev in list] [next in list] [prev in thread] [next in thread] 

List:       apache-modgzip
Subject:    Re: [Mod_gzip] Several problems with mod_gzip, apache, and PHP
From:       "Kevin Kiley" <Kiley () RemoteCommunications ! com>
Date:       2001-01-28 3:40:10
[Download RAW message or body]

Hi Matt...
This is Kevin Kiley
CTO for Remote Communications, Inc.

Comments are inline below...

-----Original Message-----
From: Matt @ FanHome.com <matt@fanhome.com>
To: mod_gzip@lists.over.net <mod_gzip@lists.over.net>
Date: Saturday, January 27, 2001 7:47 PM
Subject: [Mod_gzip] Several problems with mod_gzip, apache, and PHP


>Hi,
>
>We're running Apache 1.3.14, mod_gzip 1.3.14.6e, and PHP 4.0.4pl1 on Red
Hat
>Linux 6.2.

That could be your problem right there ( Apache 1.3.14 ).

Even if things described below 'appear' to be related to mod_gzip you
do know, don't you, that Apache 1.3.14 is totally broken and the Apache
group themselves says it was a total mistake to release it at all?

It is so full of bugs that it has never earned 'official release' status.

1.3.15 and 1.3.16 were also total screw-ups and were never released.
There is no 1.3.13 either because they messed up the tarball and
CVS forced them to bump the version without ever getting a chance
to releast 1.3.13.

1.3.17 tarball was rolled only 24 hours ago and has only even begun
to be tested but already some 'new' problems are showing up.

My suggestion is to roll back to Apache 1.3.12. It has huge, gaping
security holes but at least it was a 'stable' version.

That being said... on to your questions...

>I have output_handler = ob_gzcompress in PHP.ini

What does that do? Is PHP doing it's own compression?
If so, HOW is it doing it... using 'Content-encoding: gzip'
or streaming the data with 'Transer-encoding: chunked'
and no Content-length? Which?

> and the following pasted
>into the bottom of my HTTPD.conf file (comments pruned for clarity):
>
>#Start GZip
>LoadModule gzip_module modules/mod_gzip.so
>
>LogFormat "%h %l %u %t \"%r\" %>s %b mod_gzip:
>%{mod_gzip_compression_ratio}npct." common_with_mod_gzip_info1
>LogFormat "%h %l %u %t \"%r\" %>s %b mod_gzip: %{mod_gzip_result}n
>In:%{mod_gzip_input_size}n
>Out:%{mod_gzip_output_size}n:%{mod_gzip_compression_ratio}npct."
>common_with_mod_gzip_info2
>
>CustomLog logs/access.log common_with_mod_gzip_info2
>
># Runtime control directives...
>mod_gzip_on Yes
>mod_gzip_do_cgi Yes
>mod_gzip_do_static_files Yes
>mod_gzip_minimum_file_size 300
>mod_gzip_maximum_file_size 0
>mod_gzip_add_vinfo Yes
>mod_gzip_verbose_debug Yes
>mod_gzip_maximum_inmem_size 60000
>mod_gzip_keep_workfiles No
>mod_gzip_temp_dir "/tmp"
>
># Dynamic items...
>mod_gzip_item_include !cgi-script
>mod_gzip_item_include !.php
>
># Static items...
>mod_gzip_item_include text/*
>mod_gzip_item_include image/*
>
>
># Exclusions... MIME types and FILE types...
>mod_gzip_item_exclude .js
>mod_gzip_item_exclude .css
>
>mod_gzip_min_http 1001
># End of MOD_GZIP Configuration Directives

You already have 'mod_gzip_verbose_debug Yes' but
I don't see 'LogLevel debug'. Add that and your Apache
error.log file will contain LOTS of information about what
is happening. Run it and send me a copy or error.log
when it doesn't seem to be doing what you think it should.

>Problem #1:
>When image compression is set to "Yes" images sent to the browser from a
PHP
>file are all messed up -- e.g. they appear as big "X"es in IE5.5 and other
>browsers.

Did the PHP thingy you loaded up above already compress the image
on the way out? What KIND of image is PHP 'building'. GIF, JPEG,
BMP, What? If PHP compressed it, did it add the proper IETF
response field like 'Content-encoding: gzip'? If not... then mod_gzip
is going to compress it AGAIN and the browser is going to puke
because even though the IETF standards say that multiple
Content-encodings can be used... it is a lie. No browser can
handle more than 1 Content-encoding at a time, despite what
the IETF says.

> If I reload the page occasionally they come back, sometimes they
>show up the first time.  It's quite annoying but we use a lot of large
>images so gzip is important for us -- unfortunately I had to turn it off.
>It seems to occur on both Netscape and IE.

News flash: GIF files are already compressed using LZ77 + a
Huffman pass ( That's all that GZIP itself does ).

You are NOT going to see much improvement in graphics files
that are already 'GZIP'ed by running them through GZIP again.

GZIP itself was never actually designed to compress graphics
at all and it is based on text compression algorithms. It was
a poor choice made by CompuServe for the GIF format itself
but there you have it. People make mistakes.

I suggest just leaving it off. Despite the fact that until there is
a way to specify which User-agents should receive compression
and which ones don't... it's a bad idea to even try it. many versions
of many browser say they are HTTP 1.1 compliant and can
'Accept-encoding: gzip' but they can only handle compressed
text... they will BLOW UP if compressed graphics arrive.
Netscape is the worst offender. There is nothing in the IETF
or HTTP 1.1 standards which allows a user-agent which says
it can 'Accept-encoding' to only do it for SOME MIME types,
( It's an all or nothing specification, according to IETF ) but
there you have it. People make mistakes.

Bottom line: We do not encourage anyone to use the currently
circulating release of mod_gzip to compress graphics.

The issues that make it 'not safe' are being addressed but in
the end there will still be some browsers that are going to
puke if you try to GZIP graphics files and send them down.

MSIE handles them the best... but still not perfectly.

>Problem #2:
>We have two web servers of identical hardware / software:
>PIII 1GHz, 1GB RAM, 18.9 GB EIDE drive

Operating system? Still Red Hat Linux 6.2 as reported above?

>Now that gzip is turned on, the web servers run out of RAM about every 30
or
>40 minutes and starts swapping.  This doesn't occur when gzip is not loaded
>so I can only conclude that I have something configured incorrectly or
there
>is something in the module that leaks memory.

There is no way you can mis-configure mod_gzip and 'cause' a
memory leak. There are no memory leaks. We run mod_gzip
all day every day here and we see no such problems like the
one you are reporting. We have never seen the problems you
are reporting nor the kind of 'top' report you show below.

>Some settings in httpd.conf:
>TimeOut = 300
>KeepAlive on
>MaxKeepAliveReqests 0
>KeepAliveTimeout 60
>MinSpareServers 20
>MaxSpareServers 100
>StartServers 70
>MaxCLients 256
>MaxRequestsPerChild 2048

Looks fine. That's some pretty hefty numbers but Apache
should be able to handle that. Must be an expensive box.

These numbers don't mean anything to mod_gzip.
It uses a minimal amount of memory for compression
( Unlike standard GZIP libraries ) and it uses 'disk'
workfiles for anything over the specified limit so that
it won't 'hog' memory. It is specifically designed to
minimize the memory consumption ( unlike other
compression apporaches like Java libs ) and there
are no known memory leaks.

Again... you seem to be the first to report any 'memory' problems
like this so perhaps it's some unique 'interplay' on your system.

>We are also running the Zend Cache and Zend Optimizer, versions 1.0 for
>both, and each PHP file makes a database (Sybase ASE 11.9.2) connection to
>another server across the network.

Okay. Fine. You must have a GAZILLION bytes of memory on that
box to be pulling that off. Even the 1GB you mention seems a little thin.

>When gzip is turned off (or not loaded) each HTTPD process in top shows
>about 10MB used.

Huh? Why does one single HTTPD process need 10 megabytes of
memory? That sounds like a huge problem right there. Shouldn't
need NEAR that much just for one stupid instance of httpd.

> As soon as it is turned on however, they start ballooning
>upwards of 15 or 16MB!  This forces me to restart apache every half-hour or
>so to clear out apache's memory usage.
>A snippet of TOP is below:
> 8:23pm  up 16:16,  1 user,  load average: 3.84, 2.02, 1.13
>196 processes: 194 sleeping, 2 running, 0 zombie, 0 stopped
>CPU states: 23.5% user,  8.1% system,  0.0% nice, 68.3% idle
>Mem:  1036288K av,  656504K used,  379784K free, 1448080K shrd,   74832K
>buff
>Swap:  136512K av,    3320K used,  133192K free                   25172K
>cached
>
>  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
> 6978 nobody     0   0 17940  17M 10164 S       0  0.0  1.7   0:02 httpd
> 6983 nobody     1   0 16740  16M 10048 S       0  0.0  1.6   0:02 httpd
> 6989 nobody     0   0 16312  15M 10476 S       0  0.0  1.5   0:01 httpd
> 6917 nobody     0   0 15996  15M 10236 S       0  0.0  1.5   0:01 httpd
> 6939 nobody    12   0 15852  15M 10052 S       0  4.4  1.5   0:02 httpd
> 6882 nobody     1   0 15752  15M 10064 S       0  0.1  1.5   0:01 httpd
> 6895 nobody     0   0 15744  15M 10336 S       0  0.0  1.5   0:01 httpd
> 6937 nobody     1   0 15680  15M 10128 S       0  0.1  1.5   0:01 httpd
> 6900 nobody     0   0 15648  15M 10172 S       0  0.0  1.5   0:01 httpd
> 6969 nobody     0   0 15608  15M  9956 S       0  0.0  1.5   0:01 httpd
> 6912 nobody     9   0 15600  15M 10040 S       0  3.4  1.5   0:01 httpd
> 6887 nobody     0   0 15520  15M 10224 S       0  0.0  1.4   0:00 httpd
> 6894 nobody     0   0 15516  15M 10220 S       0  0.0  1.4   0:02 httpd
> 6926 nobody     0   0 15500  15M 10328 S       0  0.0  1.4   0:02 httpd
> 6861 nobody     0   0 15404  15M 10036 S       0  0.0  1.4   0:01 httpd
> 6952 nobody     0   0 15376  15M 10108 S       0  0.0  1.4   0:01 httpd
> 6874 nobody     0   0 15220  14M 10200 S       0  0.0  1.4   0:01 httpd
> 6870 nobody     0   0 15156  14M 10116 S       0  0.0  1.4   0:01 httpd
> 6891 nobody     0   0 15148  14M 10208 S       0  0.0  1.4   0:01 httpd
> 6885 nobody     0   0 15100  14M 10008 S       0  0.0  1.4   0:01 httpd
> 6996 nobody     0   0 15088  14M  9944 S       0  0.0  1.4   0:01 httpd
> 6902 nobody     0   0 15076  14M 10184 S       0  0.0  1.4   0:01 httpd
> 6876 nobody     0   0 15008  14M 10188 S       0  0.0  1.4   0:01 httpd

The amount of memory used pretty much matches the 'idle' state ( 68
percent ).
I'm not sure I see the problem considering the fact that 196 processes
are running.

This is a 'bad' snapshot, in your opinion, yes?
What does a 'good' one look like with 196 processes running
each allocating at lest 10MB?

What is the CPU IDLE value from a 'good' capture.
That's the critical value for a communications server.

As mentioned earlier... when mod_gzip is running it is simply calling
some Apache API's that don't 'normally' get called. If Apache 1.3.14
itself has a memory leak around those API's ( and it supposedly does )
then maybe that's all there is to it.

What does 'TOP' say if you simply run the 'stable' Apache 1.3.12 version?

>"Problem 3" (more like an informational request):
>I tried to install HyperSpace RCTPD demo the other day, but unfortunately I
>couldn't figure it out.  How do I allow HyperSpace to link up with Apache
so
>I can evaluate if it is worth purchasing?  Also, does HyperSpace nullify
the
>need for mod_gzip, or do they compliment each other?

It's easy.

RCTPD compression server is so easy to use there isn't even a 'config'
file because it isn't need. All you need is the one single binary exectuable
and everything it needs to know is supplied on the command line.

To run the RCTPD compression sever 'in front' of Apache running on
port 80 and compress all (text/*) response data going back to an
HTTP 1.1 compliant ( and gzip capable ) browser just do this...

rctpd -p1 5010 -p2 80 -th 127.0.0.1 -sr /whatever/directory/rctpd/is/in

That's it.

Now 'proxy' your browser for 'testing' to port 5010 at whatever
actualy machine RCTPD is running on and you will be
receiving compressed responses.

To compress without using alternate 'port' just move Apache
to port 81 and then fire up RCTPD on 'the front door' ( port 80 )
like this...

rctpd -p1 80 -p2 81 -th 127.0.0.1 -sr /whatever/directory/rctpd/is/in

Apache is now the backend and compression server runs as
a transparent transaction handler on 'the front door' and will
detect compression-capable browsers and 'do the right thing'.

Apache never knows what is happening.

As far as the second part of your question goes...
HyperSpace(tm) and mod_gzip work together just fine.
They are both following IETF standards to the tee and, hence,
don't 'bother' each other.

If a response is encoded, then any other IETF compliant
compression module is supposed to 'back off' and become
transparent. That's what both HyperSpace(tm) and mod_gzip do.

We run both of them together here all day, every day... no problems.

>Thanks!
>Matt
>matt@fanhome.com

Sure.
Hope some ( any ) of the above helps.

Yours...
Kevin Kiley
CTO, Remote Communications, Inc.
http://www.RemoteCommunications.com/
http://www.RemoteCommunications.com/rctpd/ - Free IETF Encoding Server
http://www.RemoteCommunications.com/apache/ab/ - Free Enhanced ApacheBench
http://www.RemoteCommunications.com/apache/mod_gzip/ - Free Content
Acceleration module for Apache

[prev in list] [next in list] [prev in thread] [next in thread]