473,320 Members | 2,006 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

File IO question.

Hi All,

I'm hoping that someone might have some pointers or
examples on how to proceed with a solution to the
following problem:

A test application, which produces a trace file, is
being run for very long periods of time. Say 72 hours
or more.

The application is often running on older PCs that
have relatively small hard drives in comparison to
to how big the trace file can become.

The trace files tend to accumulate as they're not
always deleted once archived to a network database,
and often times a machine will crash in the middle of
a
very long test due to the hard drive filling up.

There are dozens of PCs being used for the tests so
buying bigger hard drives isn't really feasible. And
I'm guessing the hard drives would fill up eventually
anyway regardless of the size of the hard drive, the
crashes just wouldn't occur quite as often.

The nice thing is that the trace files compress quite
well.

I messed around with the mmap and file object stuff
as well as the win32 extensions thinking that I could
extract and compress the data that was being written
to
the trace file, by the application, in chuncks.

Although I was able to get it to work on a contrived
setup, it didn't work when used with the real
application.

Any hints on how to get something similar to the above
to work or recommendations on alternate solutions
would
be *greatly* appreciated.

Thanks,
Joe

__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail
Jul 18 '05 #1
3 1667
You answered your own question:
I'm guessing the hard drives would fill up eventually
anyway regardless of the size of the hard drive, the
crashes just wouldn't occur quite as often.
If this statement is true, compressing the files won't
help either as that is just the same as a larger hard
drive.

But, you could limit the size and number of trace files
and periodically throw away the oldest ones to keep the
trace files at a nearly constant size. I just can't tell
from your description exactly what is getting written to
the trace file that makes it so large (debugging info?).
Maybe you could only log errors/warnings to the trace
file so it grows more slowly?

Larry Bates
Syscon, Inc.

"J Poirier" <oo**@yahoo.com> wrote in message
news:ma*************************************@pytho n.org... Hi All,

I'm hoping that someone might have some pointers or
examples on how to proceed with a solution to the
following problem:

A test application, which produces a trace file, is
being run for very long periods of time. Say 72 hours
or more.

The application is often running on older PCs that
have relatively small hard drives in comparison to
to how big the trace file can become.

The trace files tend to accumulate as they're not
always deleted once archived to a network database,
and often times a machine will crash in the middle of
a
very long test due to the hard drive filling up.

There are dozens of PCs being used for the tests so
buying bigger hard drives isn't really feasible. And
I'm guessing the hard drives would fill up eventually
anyway regardless of the size of the hard drive, the
crashes just wouldn't occur quite as often.

The nice thing is that the trace files compress quite
well.

I messed around with the mmap and file object stuff
as well as the win32 extensions thinking that I could
extract and compress the data that was being written
to
the trace file, by the application, in chuncks.

Although I was able to get it to work on a contrived
setup, it didn't work when used with the real
application.

Any hints on how to get something similar to the above
to work or recommendations on alternate solutions
would
be *greatly* appreciated.

Thanks,
Joe

__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail

Jul 18 '05 #2
J Poirier wrote:
A test application, which produces a trace file, is
being run for very long periods of time. Say 72 hours
or more.


If the test application is a python app, you coud always replace
the appropriate
open('name', 'w')
with
bz2.BZ2File('name.bz2', 'w')
Jul 18 '05 #3
J Poirier <oo**@yahoo.com> wrote in message news:<ma*************************************@pyth on.org>...

Here is my piece of advice:

Assuming that all computers have access to the network at all the
time:

1. For your trace files use one of logger classes.
2. Restrict the size of the file to some reasonable limit.
3. Name the log files with a combination of computer name and a
timestamp
4. As soon as the limit is reached logger will notify the client that
it is about to change the file. As a reaction to this callback you can
send the file to some server (see bellow) and delete the local copy.
5. At application startup send all files to the server and delete the
local copy.

"Some server":

Depending how much you want to "invest", you have several options:

1. use shared network directory and just copy the file. The drawback:
all computers have access to this shared directory. This can be a
problem in some cases.

2. use xmlrpc server/client combination. This means that you run
SimpleXMLRPCServer or something like on a dedicated machine and embed
the xmlrpcclient into your application. You can even do
filtering/compression/decompression on the fly. XMLRPC is
transactional, so to simplify the protocol, put the file name as one
parameter and the file content as another parameter.
Hi All,

I'm hoping that someone might have some pointers or
examples on how to proceed with a solution to the
following problem:

A test application, which produces a trace file, is
being run for very long periods of time. Say 72 hours
or more.

The application is often running on older PCs that
have relatively small hard drives in comparison to
to how big the trace file can become.

The trace files tend to accumulate as they're not
always deleted once archived to a network database,
and often times a machine will crash in the middle of
a
very long test due to the hard drive filling up.

There are dozens of PCs being used for the tests so
buying bigger hard drives isn't really feasible. And
I'm guessing the hard drives would fill up eventually
anyway regardless of the size of the hard drive, the
crashes just wouldn't occur quite as often.

The nice thing is that the trace files compress quite
well.

I messed around with the mmap and file object stuff
as well as the win32 extensions thinking that I could
extract and compress the data that was being written
to
the trace file, by the application, in chuncks.

Although I was able to get it to work on a contrived
setup, it didn't work when used with the real
application.

Any hints on how to get something similar to the above
to work or recommendations on alternate solutions
would
be *greatly* appreciated.

Thanks,
Joe

__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail

Jul 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: BoonHead, The Lost Philosopher | last post by:
I think the .NET framework is great! It's nice, clean and logical; in contradiction to the old Microsoft. It only saddens me that the new Microsoft still doesn't under stand there own...
1
by: Marc Cromme | last post by:
I would like to ask a question about (good ?) style and possibilities in mixing C FILE* and C++ file streams. The background is that I want to use the C libpng library from within C++, but I...
21
by: siroregano | last post by:
Hi Everyone- I'm new to this group, and almost-as-new to asking programming questions publicly, so please forgive me if I miss a convention or two! I have a text file, around 40,000 lines...
9
by: CGW | last post by:
I asked the question yesterday, but know better how to ask it, today: I'm trying to use the File.Copy method to copy a file from a client to server (.Net web app under IIS ). It looks to me that...
22
by: petermichaux | last post by:
Hi, I'm curious about server load and download time if I use one big javascript file or break it into several smaller ones. Which is better? (Please think of this as the first time the scripts...
2
by: sani8888 | last post by:
Hi everybody I am a beginner with C++ programming. And I need some help. How can I start with this program *********** The program is using a text file of information as the source of the...
12
by: dbuchanan | last post by:
Hello, (Is this the proper newsgroup?) === Background === I am building a solution with two projects. One project is my data access layer which contains my DataSet as an xsd file. The XSD...
6
by: portCo | last post by:
Hello there, I am creating a vb application which is some like like a questionare. Application read a text file which contains many questions and display one question and the input is needed...
4
by: saytri | last post by:
Hi guys! I am making a quiz. The questions are stored in a binary file. I made an option in the quiz where a user can add a question to the binary file. Altough this works, the problem is that...
14
by: =?Utf-8?B?R2lkaQ==?= | last post by:
Hi, In my windows applicationm, i need to excute a batch file. this batch file throws some text and questions to the screen, i need to catch the standard Output, check if it's a question, in...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.