[prev in list] [next in list] [prev in thread] [next in thread] 

List:       apache-modperl
Subject:    Re: Logging to a Database ...
From:       Trevor Phillips <phillips () central ! murdoch ! edu ! au>
Date:       1999-07-29 14:21:58
[Download RAW message or body]

Hans Juergen von Lengerke wrote:
> 
> >   - Splitting the logs into separate tables, and storing repeated info
> > there. More work coding, and extracting info, but would be a lot more
> > space efficient.
> 
> Also called Normalisation :-)

"Now there's a name I haven't heared in a long time..."
^_^

[snip]

> My Database looks like this:
> 
>    ---------------
>   |               |
>   |     hit       |>-------+
>   |               |        |
>    ---------------         |
>      Y        Y            |
>    [path]  [uagent]  ...[others]
> 
> Ie. One path can occur in many hits... as I said... normalised.

Yep, this is what I was thinking along the lines of...

> If you still decide to do it realtime you may find that for example your
> path table is small enough to be entirely stored in memory without your
> httpd processes becoming too big.

Hmmmm. Real-time generation I'm currently doing via a CGI (not even a
FastCGI!) (yet!), as it's not really a high-response time page, and even
with an SQL query it takes SOME time to generate; pretty much outweighs
the CGI arc-up time... The mod_perl part of the process is the logger
(my first attempt at an ApacheLogHandler!), which needs to be nice and
lean...

As Stas Bekman mentioned, it may be easier, and even faster, to just log
to a text file, and then later load the statistics into whatever form,
be it a SQL DB or other. For the bulk of the stats this would be fine,
as I was intending to round down to the nearest day for most of the
stats as is...

> In this normalised way one hit costs me about ~55 bytes of disk storage.

It certainly is appealing. ^_^
Even processing the standard Text Logs this way would be a nicer way of
archiving the old info (altho they do gzip up nicely...)

-- 
. Trevor Phillips             -           http://jurai.murdoch.edu.au/ . 
: CWIS Technical Officer         -           T.Phillips@murdoch.edu.au : 
| IT Services                       -               Murdoch University | 
 >------------------- Member of the #SAS# & #CFC# --------------------<
| On nights such as this, evil deeds are done. And good deeds, of     /
| course. But mostly evil, on the whole.                             /
 \      -- (Terry Pratchett, Wyrd Sisters)                          /

[prev in list] [next in list] [prev in thread] [next in thread]