[prev in list] [next in list] [prev in thread] [next in thread]
List: log4j-user
Subject: RE: Question about having a distributed log4j/Chainsaw environment, receiver/appender combo
From: "Scott Deboy" <sdeboy () comotivsystems ! com>
Date: 2006-02-28 16:35:34
Message-ID: E45E5CC99D174847A32B0970834F177C38A83D () csicorp6 ! pdxcsi ! com
[Download RAW message or body]
Yes, use LogFilePatternReceiver (or write all events to a database) to get history.
Yes, it's a good idea to avoid potentially blocking appenders/receivers in \
production.
If you create a Chainsaw config file that includes a receiver AND an appender, you'll \
notice that the appender will receive all of the events that are 'received' by the \
receiver.
This capability can be exploited to pull events from one environment to another \
(LogFilePatternReceiver & FileAppender used together would cause remote log files to \
be written locally). Again, this has drawbacks because of rollover issues.
Scott Deboy
COMOTIV SYSTEMS
111 SW Columbia Street Ste. 950
Portland, OR 97201
Telephone: 503.224.7496
Cell: 503.997.1367
Fax: 503.222.0185
sdeboy@comotivsystems.com
www.comotivsystems.com
-----Original Message-----
From: Giancarlo V Martinez [mailto:gvmartin@us.ibm.com]
Sent: Tue 2/28/2006 7:16 AM
To: Log4J Users List
Cc: Log4J Users List
Subject: RE: Question about having a distributed log4j/Chainsaw environment, \
receiver/appender combo
Scott thank you so much for you help, you also helped me to think of
alternatives to my current problem, which is always good. Two real quick
questions, one maybe not so quick. You suggested we use
logfilepatternreceiver because of the ability to have an event history,
correct? Also it is better than Socket, since socket could end up blocking
correct? Just trying to summarize from what you have provided me.
Second question might not be so quick. How do you setup 'forwarding' of
events? Would I have to have log4j enabled on two servers? In the diagram
would I have to have a log4j enabled program running on the Log Server in
order for the incoming events to be forwarded to the chainsaw clients, or
can appenders and receivers be configured to forward event on their own.
Thank you,
Giancarlo V. Martinez
11501 Burnet Rd 902-4, B-020
Austin TX, 78758
Office: (512)-838-9489; T/L 678-9489; Fax: (512)-823-0035
"Scott Deboy"
<sdeboy@comotivsy
stems.com> To
"Log4J Users List"
02/27/2006 10:52 <log4j-user@logging.apache.org>
PM cc
Subject
Please respond to RE: Question about having a
"Log4J Users distributed log4j/Chainsaw
List" environment, receiver/appender
combo
Your network admin should be able to devise a plan to get you access to the
logs without having to replicate them (nfs/ssh?).
Log4j's appenders and receivers -could- be used in a single log4j
configuration file to 'forward' events from one environment to another
(events received by any receiver will processed by any appender), but I
don't think that's a great idea in your case, since the only receiver that
could work is logfilepatternreceiver, and it doesn't handle file rollovers
gracefully.
Good luck
Scott
Scott Deboy
COMOTIV SYSTEMS
111 SW Columbia Street Ste. 950
Portland, OR 97201
Telephone: 503.224.7496
Cell: 503.997.1367
Fax: 503.222.0185
sdeboy@comotivsystems.com
www.comotivsystems.com
-----Original Message-----
From: Giancarlo V Martinez [mailto:gvmartin@us.ibm.com]
Sent: Mon 2/27/2006 4:15 PM
To: Log4J Users List
Subject: RE: Question about having a distributed log4j/Chainsaw
environment, receiver/appender combo
Scott,
You hit the nail on the head when you said, "On the developer side, it
wouldn't matter where the event source is (database, log file, ssh,
network), as long as their client had network access to where the log is
stored". The problem I am having is since we are in such a secure
environment, the Client does not have network access to the event source.
We thought of a central log server, which has access to the app servers,
and which can be accessed by the clients. With the Log Server, we would
only have to create a connection from the log server to the App Servers,
through the firewall. The log files themselves would still reside locally
on the App Servers, we just wanted the Log Server to pass the logging from
the App Servers to the Clients.
Basically the question I still have is, is it possible to set up a
distributed environment for logging and have the logs reside locally on
the App Servers, and in a DB for the Stored Procedure logging? From what I
have read, I am unsure, hence the question. Would the alternative be for
the App Servers to log remotely to the Log Server, and all the logs would
reside on the Log Server? Then the clients would read the log files off of
the Log Server?
Thank you,
Giancarlo V. Martinez
11501 Burnet Rd 902-4, B-020
Austin TX, 78758
Office: (512)-838-9489; T/L 678-9489; Fax: (512)-823-0035
"Scott Deboy"
<sdeboy@comotivsy
stems.com> To
"Log4J Users List"
02/27/2006 04:57 <log4j-user@logging.apache.org>
PM cc
Subject
Please respond to RE: Question about having a
"Log4J Users distributed log4j/Chainsaw
List" environment, receiver/appender
combo
A few things:
1. You can configure Chainsaw to always load a configuration on startup
View, show application wide preferences, configuration URL
2. The configuration URL can reference a remote web site or local
For example: http://www.mycompany.com/chainsaw-config.xml or
file:///c:/chainsaw-config.xml
3. This configuration file can include definitions for multiple receivers.
See the example receiver configuration available from Chainsaw's Welcome
tab
4. You can change the configuration URL and restart Chainsaw and the new
configuration will be used
My point is that you can create any number of configuration files that
allow Chainsaw to load any combination of these logs that you may be
interested in.
You may want to create a configuration file per application generating the
logs.
On the developer side, it wouldn't matter where the event source is
(database, log file, ssh, network), as long as their client had network
access to where the log is stored (database, file, etc). Developers would
just need a web page that told them:
- To access the WebSphere logs for prodA, use
http://mycompany.com/chainsaw-configuration-proda.xml
- To access all of the WebSphere logs (prodA, B, C), use
http://mycompany.com/chainsaw-configuration-all-websphere.xml
etc, etc.
Someone (maybe someone in IT, maybe you) manages these xml configuration
files (that person configures the 'plugin' entries in the config file with
the information needed to access the log).
Should you collect log files onto one server?
Only if the need for centralized management of the logs is greater than the
degraded performance you're getting from writing to a remote log system
(remember, the complexity of accessing the logs is hidden inside the
receiver configuration files).
What's the best type of receiver? I'd say the one that can read events
that are already being generated:
- LogFilePatternReceiver
if log files are accessible from http:// or file:// URLs (tailing only
works on file:// URLs for now)
- VFSLogFilePatternReceiver
if log file sare accessible via SSH or SMB or some other Jakarta
commons-vfs supported file system - next Chainsaw update will also prompt
in a GUI for username/password if needed
- CustomSQLDBReceiver
if events are stored in a database
- SocketHubAppender
if you don't mind blocking - with SocketHubAppender the server doesn't
need to know about the IP addresses of each connected Chainsaw client
Blocking:
SocketAppender/SocketReceiver and SocketHubAppender/SocketHubReceiver use
TCP socket connections to send Java-serialized logging events over the wire
from the appender to the receiver. TCP sockets can 'block' (cause the
thread sending the data to hang) if the receiver end can't keep up with the
appender end.
Hope this helps,
Scott Deboy
COMOTIV SYSTEMS
111 SW Columbia Street Ste. 950
Portland, OR 97201
Telephone: 503.224.7496
Cell: 503.997.1367
Fax: 503.222.0185
sdeboy@comotivsystems.com
www.comotivsystems.com
-----Original Message-----
From: Giancarlo V Martinez [mailto:gvmartin@us.ibm.com]
Sent: Mon 2/27/2006 2:05 PM
To: Log4J Users List
Cc: Log4J Users List
Subject: RE: Question about having a distributed log4j/Chainsaw
environment, receiver/appender combo
Scott thank you for your reply. I decided to use a picture to best
describe what we are trying to do and why.
_______________ _______________ _______________
| | | | | |
| WEBSPHERE | | WEBSPHERE | | WEBSPHERE |
| APPLICATION | | APPLICATION | | APPLICATION |
| SERVER | | SERVER | | SERVER |
| | | | | |
_______________ _______________ _______________
\ | /
\ | /
\ | /
\ ??? | ??? / ??? - Don't know
what is the best
\ | / type of
Appender/Receiver?
\ | /
\ | /
(-------------)
_________________________________________
> Logging DB |
| | ???- Oracle DB
calls? | written to |
| LOG SERVER
> -------------------------| by stored |
| |
> procedures |
_________________________________________
> using Log4PLSQL
/ | \
> **see note 2|
/ | \
(_____________)
/ | \
/ ??? |??? \ ??? - Don't know
what is the best
/ | \ type of
Appender/Receiver?
/ | \
/ | \
__________ __________ __________
| | | | | |
| Chainsaw | | Chainsaw | | Chainsaw |
| client | | client | | client |
| | | | | |
__________ __________ __________
Basically we want our development team to be able to view the logging of
any machine in our environment from a remote location, that is why we want
to use Chainsaw. We also want to be able to view the logging of our Oracle
Stored Procedures, which gets logged to the DB in the above image. Also,
since Log4J 1.3 is still in alpha, the dev lead does not feel comfortable
utilizing it, he would prefer we use 1.2.x.
We do not know which would be the best way to setup this kind of
environment. What should we use for all the question marks (???), so that
events can be viewed from any client, for any server. Saw your three
choices but which one would be the best?
The reason why we came up with the above architecture is that, we are in a
very secured area, with very strict firewall rules. To be able to connect
to WAS, the network admin has to open op a port on each machine for each
client which may connect to it. It would be impossible for us to know all
the Chainsaw clients which may want to connect to WAS, so we thought a
middle-man, the "Log Server" would be our best bet. We thought, as stated
in my previous note, we could have an appender/receiver connection from WAS
to the Log Server, this connection would also be active, so that would
handle the issue of event history.
Correct me if I am wrong, but if the connection is always active then no
events should be lost, correct? What would be the best way for multiple
Chainsaw Clients to view the events from multiple WebSphere Servers?
Second, the DBA does not want us to use log4j on the Logging DB. He does
not want an instance of Java running on that machine. What would be the
best way for Chainsaw clients to view the events of the Stored Procedures,
could we also route those requests through the Log Server? I know
Log4PLSQL is not in the domain of this forum, but if you could nudge us in
the right direction it would be great.
For the record I am very new to log4J and chainsaw so some of your
responses I could not follow. What did you mean by " -blocking-
network-based appenders"? How can the App Server be "blocked"?
Also I didn't understand this sentence, "The Chainsaw config can be
managed by IT and accessed via URL, so if you use log files, it doesn't
matter where they are because everyone uses a single configuration file to
access the logs", how would chainsaw view these files, via ssh? I don't
think would work however due to our network constraints.
Thank you so much for your help,
Giancarlo V. Martinez
11501 Burnet Rd 902-4, B-020
Austin TX, 78758
Office: (512)-838-9489; T/L 678-9489; Fax: (512)-823-0035
"Scott Deboy"
<sdeboy@comotivsy
stems.com> To
"Log4J Users List"
02/27/2006 10:49 <log4j-user@logging.apache.org>
AM cc
Subject
Please respond to RE: Question about having a
"Log4J Users distributed log4j/Chainsaw
List" environment, receiver/appender
combo
Do you need event history? (events created prior to when you started
Chainsaw)? If so, that leaves out the socket-based appenders (at least in
1.2.x).
SocketHubAppender and SocketHubReceiver are not a bad way to go if you
don't need event history, but I wouldn't use -blocking- network-based
appenders in a server app. Even wrapping them in AsyncAppender isn't
sufficient to prevent possible blocking of the app server if Chainsaw is on
a slow link (at least in log4j 1.2.x).
Assuming you're using log4j 1.2.x in the server environment, I'd use one of
these combinations:
- FileAppender and LogFilePatternReceiver or VFSLogFilePatternReceiver (if
over ssh)
- JMSAppender and custom code to write the events to a database and
CustomSQLDBReceiver
- JDBCAppender (http://www.dankomannhaupt.de/projects/) and
CustomSQLDBReceiver
The Chainsaw config can be managed by IT and accessed via URL, so if you
use log files, it doesn't matter where they are because everyone uses a
single configuration file to access the logs (no real need for a
centralized server to get the same effect).
When you can use log4j 1.3 in your environment on the appender-side, you
could use:
- DBAppender and DBReceiver (if you don't mind using log4j's schema for
logging events)
- An enhancement to SocketHubAppender that allows you to specify the
number of events that are buffered - the buffered events are provided to
new SocketHubReceiver clients
- Two -nonblocking- network appenders and receivers: UDPAppender/Receiver
and MulticastAppender/Receiver
Log files are probably the best way to go for now - and the latest version
of VFSLogFilePatternReceiver will allow you to access logs accessible via
ssh without you having to embed the username/password in the config file
(the next update to Chainsaw will support a GUI login window for
VFSLogFilePatternReceiver).
I hope this helps, feel free to ask more questions.
Scott Deboy
COMOTIV SYSTEMS
111 SW Columbia Street Ste. 950
Portland, OR 97201
Telephone: 503.224.7496
Cell: 503.997.1367
Fax: 503.222.0185
sdeboy@comotivsystems.com
www.comotivsystems.com
-----Original Message-----
From: Giancarlo V Martinez [mailto:gvmartin@us.ibm.com]
Sent: Mon 2/27/2006 7:43 AM
To: log4j-user@logging.apache.org
Subject: Question about having a distributed log4j/Chainsaw environment,
receiver/appender combo
I have been asked by my company if it is possible to setup a distributed
environment for log4j and Chainsaw, in which all of our application servers
would be logging to one "Log Server", and multiple Chainsaw clients would
connect to the "Log Server". That way any user can connect to a central
server and view the logging of all the application servers.
I did some research and I think I need to setup the following environment:
- A SocketAppender on each of the Application Servers
- A SocketReciever on the "Log Server" for each appender/receiver combo
- A SocketHubAppender on the "Log Server" so that multiple chainsaw clients
can connect
- A SocketHubReceiver for each of the Chainsaw clients.
I was wondering if this was even possible, to have logging messages routed
through a server with a SocketReceiver/SocketHubAppender Combination, and
if so how would I set this up? I could I get the two interfaces to
communicate with each other?
Thank you,
Giancarlo V. Martinez
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic