473,721 Members | 4,051 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Efficient 'logging' mechanism?

Hi,

I have an application which sends/receives messages at a very high
rate. This app needs to 'log' the contents of these messages. Since the
msg rate is high, 'logging' each and every msg to a disk-file reduces
the app's performace significantly. I want to device a 'logging'
mechanism where the 'log' is actually buffered and it will be written
to the disk periodically. The time of writing to disk should be
controlled by the app rather than by the OS. At times, I also want to
reset the in-memory log without writing to disk.

Currently I use an 'ofstream' object for this purpose. But it does not
allow me to *control* buffering. I tried out another option as follows.

I extended the MFC's CMemFile class.

class CMemLogFile : public CMemFile
{
public:
CMemLogFile(CSt ring sName);
virtual ~CMemLogFile();

// Base class overrides
void Close();

void Truncate();
void WriteToDisk();

private:
CFile m_file;
CString m_sName;
};

CMemLogFile::CM emLogFile(CStri ng sName)
{
m_sName = sName;
m_file.Open(sNa me, CFile::modeCrea te|CFile::modeW rite);
}

void CMemLogFile::Cl ose()
{
m_file.Close();
CMemFile::Close ();
}

void CMemLogFile::Tr uncate()
{
this->SetLength(0) ;
}

void CMemLogFile::Wr iteToDisk()
{
char buf[READ_LEN] = "";
this->SeekToBegin( );

int iBytesRead = 0;

do
{
iBytesRead = this->Read(buf, READ_LEN);
m_file.Write(bu f, iBytesRead);

} while ( iBytesRead >= READ_LEN);

m_file.Flush();
}

This works fine except for the fact that I don't have the advance
formatting options such as setw() that ofstream provides.

My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)

2. Failing (1), is there a way that we can increase the buffer size of
an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLengt h(0)
truncates)

Jul 22 '05 #1
6 5443
<id******@eudor amail.com> wrote in message
Currently I use an 'ofstream' object for this purpose. But it does not
allow me to *control* buffering. I tried out another option as follows.
If you want to increase the length of the buffer, take a look at function
streambuf::setp (char * begin, char * end). If I remember correctly, the
streambuf does not own the buffer, so you're responsible for deleting the
memory (either by calling delete[], using std::vector<cha r>, etc).
My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)
What kind of control are you after? The endl function flushes the stream,
that is writes the buffer to the actual output file. If nounitbuf is on (or
off, I don't remember), then every write will always flush the buffer.
2. Failing (1), is there a way that we can increase the buffer size of
an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLengt h(0)
truncates)


One calls functions like ostream::setp.
Jul 22 '05 #2
Hi,
I think you should put two threads, one would be the main thread for
processing the message and other would create the log file. Make one
queue in each of the thread. As soon as you get the message push it to
the queue of both the threads. Or you can use the command pattern, as
you like.. I think this will substantially reduce the pressure from
the main thread.

Saurabh Aggrawal
Sr. S/w Programmer

Jul 22 '05 #3

id******@eudora mail.com wrote:
Hi,

I have an application which sends/receives messages at a very high
rate. This app needs to 'log' the contents of these messages. Since the msg rate is high, 'logging' each and every msg to a disk-file reduces
the app's performace significantly. I want to device a 'logging'
mechanism where the 'log' is actually buffered and it will be written
to the disk periodically. The time of writing to disk should be
controlled by the app rather than by the OS. At times, I also want to
reset the in-memory log without writing to disk.

Currently I use an 'ofstream' object for this purpose. But it does not allow me to *control* buffering. I tried out another option as follows.
I extended the MFC's CMemFile class.

class CMemLogFile : public CMemFile
{
public:
CMemLogFile(CSt ring sName);
virtual ~CMemLogFile();

// Base class overrides
void Close();

void Truncate();
void WriteToDisk();

private:
CFile m_file;
CString m_sName;
};

CMemLogFile::CM emLogFile(CStri ng sName)
{
m_sName = sName;
m_file.Open(sNa me, CFile::modeCrea te|CFile::modeW rite);
}

void CMemLogFile::Cl ose()
{
m_file.Close();
CMemFile::Close ();
}

void CMemLogFile::Tr uncate()
{
this->SetLength(0) ;
}

void CMemLogFile::Wr iteToDisk()
{
char buf[READ_LEN] = "";
this->SeekToBegin( );

int iBytesRead = 0;

do
{
iBytesRead = this->Read(buf, READ_LEN);
m_file.Write(bu f, iBytesRead);

} while ( iBytesRead >= READ_LEN);

m_file.Flush();
}

This works fine except for the fact that I don't have the advance
formatting options such as setw() that ofstream provides.

My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)

2. Failing (1), is there a way that we can increase the buffer size of an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLengt h(0) truncates)


Don't try to play around with the internals of ofstream: you're
likely to move into a area of C++ where most of the things are
implementation specific.
An alternative approach is to provide the necessary output
operators in the class CMemLogFile itself and temporary write
to std::ostringstr eam member

<pre>
class CMemLogFile {
public:

template<typena me T> Logfile& operator<<(cons t T& t);
Logfile& operator<<(std: :ostream& (*func)(std::os tream&));
Logfile& operator<<(std: :ios& (*func)(std::io s&));
Logfile& operator<<(std: :ios_base& (*func)(std::io s_base&));

private:
std::ostringstr eam buffer_;
};

Logfile& Logfile::operat or<<(const T& t)
{
buffer_ << t;
return *this;
}

Logfile::operat or<<(std::ostre am& (*func)(std::os tream&))
{
buffer_ << func;
return *this;
}
</pre>

and similar for the 2 other output operators.

Hope this helps,
Stephan Brönnimann
br****@osb-systems.com
http://www.osb-systems.com
Open source rating and billing engine for communication networks.

Jul 22 '05 #4
Siemel Naran wrote:
<id******@eudor amail.com> wrote in message
Currently I use an 'ofstream' object for this purpose. But it does not allow me to *control* buffering. I tried out another option as follows.

If you want to increase the length of the buffer, take a look at function streambuf::setp (char * begin, char * end).
'setp()' is a protected member function and not intended for user
access.
In fact, I would expect that a file buffer will fail with some rather
strange problems if you start fiddling with its internal buffering. I
guess you ment 'setbuf()' and 'pubsetbuf()'. Note, however, that these
functions are not required to have any effect except that
'pubsetbuf(0, 0)' is supposed to turn off buffering for file buffers.
If I remember correctly, the
streambuf does not own the buffer, so you're responsible for deleting the memory (either by calling delete[], using std::vector<cha r>, etc).
If you are referring to 'pubsetbuf()', this is true. However, I
consider
this function to be mostly useless. In particular, it is useless with
respect to controlling the buffer size (e.g. my implementation
deliberately
ignores any passed buffer).
My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and the default state 'nounitbuf' doesn't seem to work)


What kind of control are you after? The endl function flushes the

stream, that is writes the buffer to the actual output file. If nounitbuf is on (or off, I don't remember), then every write will always flush the

buffer.

If 'unitbuf' is set the stream flushes the the buffer after each
operation
(i.e. when the internal 'sentry' object is destructed).
2. Failing (1), is there a way that we can increase the buffer size of an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLengt h(0) truncates)


One calls functions like ostream::setp.


There is no member 'setp()' in 'ostream'. There is, however, a
protected
member 'setp()' in stream buffers.

I would implement a specialized stream buffer for this which just
buffers
the output until it is explicitly flushed. If accidental 'endl's should
also not flush the buffer, flushing could be triggered using a separate
function. Of course, the stream buffer would use a file stream buffer
underneath.
--
<mailto:di***** ******@yahoo.co m> <http://www.dietmar-kuehl.de/>
<http://www.contendix.c om> - Software Development & Consulting

Jul 22 '05 #5

id******@eudora mail.com wrote:
Hi,

I have an application which sends/receives messages at a very high
rate. This app needs to 'log' the contents of these messages. Since the msg rate is high, 'logging' each and every msg to a disk-file reduces
the app's performace significantly. I want to device a 'logging'
mechanism where the 'log' is actually buffered and it will be written
to the disk periodically. The time of writing to disk should be
controlled by the app rather than by the OS. At times, I also want to
reset the in-memory log without writing to disk.

Currently I use an 'ofstream' object for this purpose. But it does not allow me to *control* buffering. ...
Incorrect assumption.
[...]
My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)
Yes. Insert your own buffer mechanism between the ofstream and its
std::streambuf. The member you need is .rdbuf, use the old one
as the output destination for your own streambuf.
3. How can you reset an ofstream buffer? (Just like CFile::SetLengt h(0) truncates)


It's your own object. Just like you want to. Since you probably will
keep a std::vector<cha r> in it, it's probably vector<char>::c lear().
Regards,
Michiel Salters

Jul 22 '05 #6
"Dietmar Kuehl" <di***********@ yahoo.com> wrote in message
Siemel Naran wrote: If you are referring to 'pubsetbuf()', this is true. However, I
consider
this function to be mostly useless. In particular, it is useless with
respect to controlling the buffer size (e.g. my implementation
deliberately
ignores any passed buffer).
So what's the point of the function?
One calls functions like ostream::setp.


There is no member 'setp()' in 'ostream'. There is, however, a
protected
member 'setp()' in stream buffers.


Oops, I mean seekp.
I would implement a specialized stream buffer for this which just
buffers
the output until it is explicitly flushed. If accidental 'endl's should
also not flush the buffer, flushing could be triggered using a separate
function. Of course, the stream buffer would use a file stream buffer
underneath.


How would you distinguish o << flush from o.flush()?
Jul 22 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
3799
by: JT | last post by:
i have a text file that is used as a logfile for easier debugging. i am using the OpenTextFile method to write to this file. this method exists inside a logging function that can be called from anywhere in my ASP. this works great for our development system b/c we have a limited number of users. however, we're unable to use this logging mechanism on live due to concurrency issues. when two users attempt to open and write to the text...
1
418
by: Gram | last post by:
Hello, Can anoyone help me with the following: I have a application online using ASP in which users log in and log out. However, a lot of users simply close the window, bypassing my log-out script. I have tried using the Javascript onUnload function with success,but refreshing the screen also triggers this at the wrong time, Has anyone got any suggestions? Thanks in advance,
0
343
by: idesilva | last post by:
Hi, I have an application which sends/receives messages at a very high rate. This app needs to 'log' the contents of these messages. Since the msg rate is high, 'logging' each and every msg to a disk-file reduces the app's performace significantly. I want to device a 'logging' mechanism where the 'log' is actually buffered and it will be written to the disk periodically. The time of writing to disk should be controlled by the app rather...
6
7321
by: pmatos | last post by:
Hi all, I am trying to create a simple but efficient C++ logging class. I know there are lots of them out there but I want something simple and efficient. The number one requirement is the possibility of shutting logging down at compile time and suffer no performance penalty whatsoever for getting logging on whenever I wish. Of course that I would need to recompile the project each time I want to turn logging on or off. But given a...
6
9967
by: Kevin Jackson | last post by:
Let's say we log exceptions to the windows application event log using the Exception Management Application Block. Is there a pattern that can be used so the exception is logged only once and not everytime up through the call chain? Is there anything a caller can key off of to check to see if it should also log the exception?
16
2169
by: Einar Høst | last post by:
Hi, I'm getting into the Trace-functionality in .NET, using it to provide some much-needed logging across dlls in the project we're working on. However, being a newbie, I'm wondering if some more experienced loggers can provide me with some ideas as to how to log in a simple yet flexible manner. For instance, I'd like the code to be as uncluttered as possible by Trace statements. As an example of basic logging functionality, I've come...
0
1489
by: vineetkumar | last post by:
dear all, i have a problem to maintain the Transaction Logging Mechanism in postgresql 8.1 so please send reply how i can maintain Rollback Segment, LogFiles, and every day transaction file . is there any facility in postgres because oracle provide archive
9
1954
by: Rasika WIJAYARATNE | last post by:
Hi guys, Please check this out: http://rkwcoding.blogspot.com/2007/07/error-logging.html
6
5686
by: Vyacheslav Maslov | last post by:
Hi all! I have many many many python unit test, which are used for testing some remote web service. The most important issue here is logging of test execution process and result. I strongly need following: 1. start/end timestamp for each test case (most important) 2. immediate report about exceptions (stacktrace) 3. it will be nice to use logging module for output
0
8736
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9373
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9227
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9077
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8019
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6676
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
1
3202
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2588
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2137
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.