Enterprise Library - Logging performances

[Udi]

Hi All,
I'm developping an application that has a very strong logging
requirement -
we should be able to log up to 5000 messages per second, most of them
are reletively small -(16 - 30 Words) while few of them might get to
1Kb in size. The data is binary (numbers) that should be converted to
text of course.

I'm affraid of a "logging-bottleneck" in my app.

Do you know if the entlib can/can't handle this kind of thruput?
Do you think I should use asynch logging (MSMQ?) ?
Do you have any other suggestions/recommendations for me?
I'd appreciate it if you could share any experience you had with the
logging application block with similar requirements.

Thanks!
Udi.

[jeremiah johnson]

that is an awesome logging requirement. awesome as in it "inspires awe."

i don't know that there is a bottleneck in the existing logging
libraries, you'll need to performance test in your environment for that,
but based on experience i've had in the past, you're probably going to
need a logging server.

i'm very interested to see what your results are. if you could, and if
you're going to do performance testing, please let me (or the group)
know what your results are, and how you tested.

in particular i'd love to know just how much you could log to the
network, as well as how much you could log to disk and still keep a
useful system.

thanks.

Udi wrote:

Hi All,
I'm developping an application that has a very strong logging
requirement -
we should be able to log up to 5000 messages per second, most of them
are reletively small -(16 - 30 Words) while few of them might get to
1Kb in size. The data is binary (numbers) that should be converted to
text of course.

I'm affraid of a "logging-bottleneck" in my app.

Do you know if the entlib can/can't handle this kind of thruput?
Do you think I should use asynch logging (MSMQ?) ?
Do you have any other suggestions/recommendations for me?
I'd appreciate it if you could share any experience you had with the
logging application block with similar requirements.

Thanks!
Udi.

[jeremiah johnson]

I should also mention that I've personally performance tested some
pretty incredible data generation systems, to see just how long it takes
to read and write data from a disk on my desktop.

video games and their load times have given people the completely wrong
impression of disk performance on Windows machines. I was able to write
out 100Mb of data to disk in under two seconds, and read it all into
memory in less than one.

based on this, if your logging application bottlenecks you, I would
consider writing a custom lightweight function or class to do your
logging. You desktop or server should certainly be able to handle 5000
1k messages per second.

but as i said before, I personally do not know anything of any built-in
logging system in C#, what its performance would be, and I would be very
interested in finding your results.

jeremiah

Udi wrote:

Hi All,
I'm developping an application that has a very strong logging
requirement -
we should be able to log up to 5000 messages per second, most of them
are reletively small -(16 - 30 Words) while few of them might get to
1Kb in size. The data is binary (numbers) that should be converted to
text of course.

I'm affraid of a "logging-bottleneck" in my app.

Do you know if the entlib can/can't handle this kind of thruput?
Do you think I should use asynch logging (MSMQ?) ?
Do you have any other suggestions/recommendations for me?
I'd appreciate it if you could share any experience you had with the
logging application block with similar requirements.

Thanks!
Udi.

[sloan]

The Enterprise Library gives you the flexibility to write your own
Logging.

I'd consider that,.. and write a custom logger .. and consider hooking
that into MSMQ.

It all depends. If you're writing out txt files (a new one for each
event) and the files are small, then I wouldn't go to the hassle of
msmq.

If there is anything which needs more then 100ms of time (Just a number
I pulled out of the air), then I'd look at MSMQ.

...

I have one custom logger , that creates a new txt file for each unique
IP address..AND each hour.

192_168_100_101_04_10_2006_11_AM_HR.txt

something like that. I write entries in there.

Then I go and read them, and put them into a db at a later time.
I do this specfically to not incur the db penalty... as I'm trying to
uber max out my scaleability.

But .. on antoher project, I log stuff right to the db(). Its a
decision you need to make after weighing everything.


Udi wrote:

Hi All,
I'm developping an application that has a very strong logging
requirement -
we should be able to log up to 5000 messages per second, most of them
are reletively small -(16 - 30 Words) while few of them might get to
1Kb in size. The data is binary (numbers) that should be converted to
text of course.

I'm affraid of a "logging-bottleneck" in my app.

Do you know if the entlib can/can't handle this kind of thruput?
Do you think I should use asynch logging (MSMQ?) ?
Do you have any other suggestions/recommendations for me?
I'd appreciate it if you could share any experience you had with the
logging application block with similar requirements.

Thanks!
Udi.