Concurrent Logging: How?

[Ravi Shankar]

Hi all,

I have an enterprise application. I am using Apache Log4J for the logging
purposes. WHen a request is received by the application, it goes throug
servlets and many classes, and the necessary details are logged to files,
which has autorate option. I mean when certain file exceeds the size,
automatically another file is generated for a maximum of 10 files and then
again starts from the first.

So far is so good. Now assume that say 100 users are concurrently making a
request. Since servlets are mutithreaded, the logs generated does not show
each request sequentially. All the logs of all the requests are mixed
together making it difficult and highly spagettic to read and understand.

How can I make the logging strategy such that each user request is written
in a sequential manner so that there are no overlaps of the logs? Please
advise, thanks
Best regards,
Ravi

[Daniel Parker]

"Ravi Shankar" <su*********@pacific.net.sg> wrote in message
news:ck**********@nobel.pacific.net.sg...

So far is so good. Now assume that say 100 users are concurrently making a
request. Since servlets are mutithreaded, the logs generated does not show
each request sequentially. All the logs of all the requests are mixed
together making it difficult and highly spagettic to read and understand.

How can I make the logging strategy such that each user request is written
in a sequential manner so that there are no overlaps of the logs? Please
advise, thanks

Isn't this more an issue of having an appropriate viewer for the log?
Assuming you create log entries that identify the user, by session id or
whatever, you just need a viewer that will group them appropriately. If you
take this strategy, you will of course need to format the log entries
consistently. The alternative strategy, of grouping the messages before
they are written to the log, would be bad because generally you want to
write the messages to a physical medium as soon as possible, for obvious
reasons. You could write the messages to a database, in which case queries
on user would be easier, but there's an obvious downside to that as well.
So my advice: format your messages consistently, include a user field, and
implement an appropriate viewer.

Regards,
Daniel Parker

[Raymond DeCampo]

Daniel Parker wrote:

"Ravi Shankar" <su*********@pacific.net.sg> wrote in message
news:ck**********@nobel.pacific.net.sg...

So far is so good. Now assume that say 100 users are concurrently making a
request. Since servlets are mutithreaded, the logs generated does not show
each request sequentially. All the logs of all the requests are mixed
together making it difficult and highly spagettic to read and understand.

How can I make the logging strategy such that each user request is written
in a sequential manner so that there are no overlaps of the logs? Please
advise, thanks

Isn't this more an issue of having an appropriate viewer for the log?
Assuming you create log entries that identify the user, by session id or
whatever, you just need a viewer that will group them appropriately. If you
take this strategy, you will of course need to format the log entries
consistently. The alternative strategy, of grouping the messages before
they are written to the log, would be bad because generally you want to
write the messages to a physical medium as soon as possible, for obvious
reasons. You could write the messages to a database, in which case queries
on user would be easier, but there's an obvious downside to that as well.
So my advice: format your messages consistently, include a user field, and
implement an appropriate viewer.

log4j has a few tools to help you to this end. You can use NDC or MDC
to associate the user to the thread; that way you do not need to change
all of your existing log statements. Also, if you write the log in XML
format (there is a formatter in log4j for this) it can be parsed by
Chainsaw (a GUI tool from log4j) giving you a powerful viewer. You can
also configure log4j to send messages directly to Chainsaw via a socket.

For more details and assistance I suggest you refer to the log4j-users
mailing list from apache.org.

HTH,
Ray

--
XML is the programmer's duct tape.

[Calum Grant]

Ravi Shankar wrote:

Hi all,

I have an enterprise application. I am using Apache Log4J for the logging
purposes. WHen a request is received by the application, it goes throug
servlets and many classes, and the necessary details are logged to files,
which has autorate option. I mean when certain file exceeds the size,
automatically another file is generated for a maximum of 10 files and then
again starts from the first.

So far is so good. Now assume that say 100 users are concurrently making a
request. Since servlets are mutithreaded, the logs generated does not show
each request sequentially. All the logs of all the requests are mixed
together making it difficult and highly spagettic to read and understand.

How can I make the logging strategy such that each user request is written
in a sequential manner so that there are no overlaps of the logs? Please
advise, thanks

You'd need to write a unique request-id in each line of the log. Then
you can apply filters (e.g. grep, perl) to extract useful information
from the log. You could write (or get someone else to write) a script
to process these log-files and collate requests.

You can also use SQL for logging, so then it's slightly easier to apply
queries and filters, for example to order by (request-id, time) instead
of just time.

Calum

[scott]

The latest version of Chainsaw V2 can help you once you've added
unique identifiers to each event (via MDC) to slice & dice logs - you
can even combine client and server logs together.

Chainsaw V2 is available via WebStart here:
http://logging.apache.org/log4j/docs/chainsaw.html

Chainsaw can display and tail a log file generated by any logging
framework using LogFilePatternReceiver.

Here's a link to the LogFilePatternReceiver javadoc, describing the
'logFormat' keywords:
http://cvs.apache.org/viewcvs.cgi/lo...1.19&view=auto

If you want to tail your log file, modify the below-provided xml as
needed and save as chainsaw.xml.

Inside Chainsaw, select:
view-show application wide preferences
- enter the URL of this chainsaw.xml in the 'automatic configuration
URL' field
- restart chainsaw
chainsaw.xml:
------------------------------
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"
debug="true">

<plugin name="logFileReceiver"
class="org.apache.log4j.varia.LogFilePatternReceiv er">
<param name="fileURL" value="file:///c:/downloads/log4j/sample.log" />
<param name="timestampFormat" value="yyyyMMddHHmmssSSS"/>
<param name="logFormat" value="TIMESTAMP LEVEL(FILE:LINE)#CLASS,LOGGER
THREAD!MESSAGE"/>
<param name="name" value="logfileReceiver" />
<param name="tailing" value="true" />
</plugin>
<root>
<level value="debug"/>
</root>
</log4j:configuration>
------------------------------
Hope this helps - there is a tutorial available from the 'Welcome' tab
that explains how to use it.

Feel free to send an email to the log4j user mailing list with
questions:
http://logging.apache.org/site/mailing-lists.html

Scott
Calum Grant <ca****@no.onetel.spam.com> wrote in message news:<_P**************@newsfe6-gui.ntli.net>...

Ravi Shankar wrote:

Hi all,

I have an enterprise application. I am using Apache Log4J for the logging
purposes. WHen a request is received by the application, it goes throug
servlets and many classes, and the necessary details are logged to files,
which has autorate option. I mean when certain file exceeds the size,
automatically another file is generated for a maximum of 10 files and then
again starts from the first.

So far is so good. Now assume that say 100 users are concurrently making a
request. Since servlets are mutithreaded, the logs generated does not show
each request sequentially. All the logs of all the requests are mixed
together making it difficult and highly spagettic to read and understand.

How can I make the logging strategy such that each user request is written
in a sequential manner so that there are no overlaps of the logs? Please
advise, thanks

You'd need to write a unique request-id in each line of the log. Then
you can apply filters (e.g. grep, perl) to extract useful information
from the log. You could write (or get someone else to write) a script
to process these log-files and collate requests.

You can also use SQL for logging, so then it's slightly easier to apply
queries and filters, for example to order by (request-id, time) instead
of just time.

Calum

[Ravi Shankar]

HI all,

Thanks a lot to all for the wonderful help. I will be using Chainsaw with
NDC/MDC.

Thanks and regards,
Ravi

"Ravi Shankar" <su*********@pacific.net.sg> wrote in message
news:ck**********@nobel.pacific.net.sg...

Hi all,

I have an enterprise application. I am using Apache Log4J for the logging
purposes. WHen a request is received by the application, it goes throug
servlets and many classes, and the necessary details are logged to files,
which has autorate option. I mean when certain file exceeds the size,
automatically another file is generated for a maximum of 10 files and then
again starts from the first.

So far is so good. Now assume that say 100 users are concurrently making a
request. Since servlets are mutithreaded, the logs generated does not show
each request sequentially. All the logs of all the requests are mixed
together making it difficult and highly spagettic to read and understand.

How can I make the logging strategy such that each user request is written
in a sequential manner so that there are no overlaps of the logs? Please
advise, thanks
Best regards,
Ravi