473,320 Members | 2,088 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Cropping log files

Yet another possible newbie question.

I'm tyring to figure out how to best crop a log file. I have a file
that grows infinitely in length. I want to keep only the last n entries
in the file.

I've been looking for some sort of ascii-database or log file management
python module that lets me say: how many records in the file? and then
say: delete the first nr - n records.

No joy.

I don't want to suck the whole file into memory if I can help it, and I
can't help thinking that doing a
for line in file(logfile)
nr += 1

to count the number of lines, then reading the file again, discarding
the first nr - n records, writing the rest to a temp file, and then
renaming the files is the most efficient way to go.

Not only that, but what happens if the logfile is written to while I'm
doing all of this - I may lose log file entries....

I found FLAD, even that seems to be overkill for what I need. So, any
modules out there for log file management?

The logging module lets me log events, but there aren't any tools for
managing the log files created in the way I need.... The
RotatingFileHandler rotates logs in the sense of logrotate, but what I
need is to keep only the last n records in one file.

-Kamus

--
o__ | If you're old, eat right and ride a decent bike.
,>/'_ | Q.
(_)\(_) | Usenet posting`

Jul 18 '05 #1
6 2703
If you need a ASCII database, you could have a look at kirbybase
--
Rony

/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
/ bu************@yahoo.fr (delete _no_spam)
/
| www.bucodi.com - My work
\ www.ifrance/karamusique -- My hobby
\_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Jul 18 '05 #2
Kamus of Kadizhar <ya*@NsOeSiPnAeMr.com> wrote in
news:br***********@news3.infoave.net:
The logging module lets me log events, but there aren't any tools for
managing the log files created in the way I need.... The
RotatingFileHandler rotates logs in the sense of logrotate, but what I
need is to keep only the last n records in one file.


Keeping a fixed number of records in one file is difficult if the records
aren't a fixed length, which is why most logging systems limit the number
of records that are kept by the size of the data, rather than by the number
of records.

If you can modify your requirements to say that you need to keep 'at least
n records' then rotating file logger will do that, just make sure that the
size that is kept in all files is at least n times the largest record you
might log. Of course the records will then be split across multiple files.

If you really need to keep n records in one file, then the best thing is
probably to write your own handler for the logging module and store the log
records in a fixed length format in the file.

i.e. you choose a size larger than your largest log record and pad each log
record to that length with nulls. Then you just write out to the file as
though it were a circular buffer. You need some way of finding the first
record again: one way is to always ensure you have at least one empty
record in the file so when you first open the file you scan to the first
empty record, then each time you write to the file you write the new record
followed by a new empty record, then skip back so the empty record will be
overwritten. When you reach record number n+1 in the file you simply wrap
back to the start. You'll also want to write code to read a logfile
starting after the last empty record.

If you can't set a hard upper bound on the size of a record you will need a
more complicated file structure, or if that 'n' doesn't absolutely have to
be set in stone simply allow occasional long records to wrap over into the
next record and handle that case in both writing and reading code.

The NTEventHandler, (for those of us on Windows systems) does pretty much
this last option, it uses event logs files which are a fixed length and
wrap round, but individual records can span more than one block in the
file, so the number of entries in a logfile is actually an upper limit.

--
Duncan Booth du****@rcp.co.uk
int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
"\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?
Jul 18 '05 #3
Duncan Booth wrote:

If you can modify your requirements to say that you need to keep 'at least
n records' then rotating file logger will do that, just make sure that the
size that is kept in all files is at least n times the largest record you
might log. Of course the records will then be split across multiple files.


You know, that just might work. I need more-or-less n records; the
number is not critical.

So, if I allow the rotating logger a max file size that accomodates
approx. n/4 records, tell it to keep older files to a depth of 6, then
read older log files until I get some number of records greater than n....

I just have bad experience with letting log files get to infinite
length... :-)

--
o__ | If you're old, eat right and ride a decent bike.
,>/'_ | Q.
(_)\(_) | Usenet posting`

Jul 18 '05 #4
"Kamus of Kadizhar" <ya*@NsOeSiPnAeMr.com> wrote in message
news:br***********@news3.infoave.net...
....
I'm tyring to figure out how to best crop a log file. I have a file
that grows infinitely in length. I want to keep only the last n entries
in the file. ....

An alternative solution may be to write date-based files with deleting of
files older than some limit.
Here is an example (class HourlyLog) of date-based hourly logging with
day-based cleanup:

------------------------------------------------------
import glob, os, time

def Now(): # UTC time now as a string
return time.asctime(time.gmtime())

def DD_HH(): # 2 digit utc day & hour
return time.strftime("%d_%H",time.gmtime())

def Hhmmss(): # "hh:mm:ss" string
return time.strftime("%H:%M:%S",time.gmtime())

def DaysAgo(number): # time days ago
return time.time() - (86400 * number)

class Hour:
def __init__(self):
self.hour = time.gmtime()[3]

def change(self):
hour = time.gmtime()[3]
if hour != self.hour:
self.hour = hour
return 1
else:
return 0

# filename must contain '##_##' for the 2 digit day and hour fields
def FileHourLog(f,s): # append string to daily logfile (nb: win32 does
lf -> crlf)
try:
ff=open(DD_HH().join(f.split('##_##',1)),'aU')
except:
ff=open(DD_HH().join(f.split('##_##',1)),'wU')
ff.write(''.join(['-----< ',Now(),'-----','\n',s.replace('\r\n','\r').replace('\r','\n'),' \n']))

ff.close()

def FileBefore(f,time): # return true if file modification date is
before time()-secs
try:
if os.stat(f)[8] < time:
return 1
else:
return 0
except:
return 0 # may not exist in multi-threaded app.

def FileTidy(list,at): # delete files older than time
for file in list:
try:
if FileBefore(file,at):
os.remove(file)
except:
pass # may not exist in multi-threaded app.

def FilesMatching(path,pattern): # return a list of files matching pattern.
return glob.glob(os.path.join(path,pattern))

class HourlyLog: # filename as per FileLog; number of days to keep
def __init__(self,f,n):
self.hour = Hour()
self.f = f
self.n = n

def log(self,s):
FileHourLog(self.f,s)
if self.hour.change():
[path, name] = os.path.split(self.f)
pattern = name.replace('#','?')
folder = FilesMatching(path,pattern)
FileTidy(folder,DaysAgo(self.n))
-------------------------------------------------------------

Colin Brown
PyNZ
Jul 18 '05 #5
Kamus of Kadizhar <ya*@NsOeSiPnAeMr.com> wrote:
Yet another possible newbie question.

I'm tyring to figure out how to best crop a log file. I have a file
that grows infinitely in length. I want to keep only the last n entries
in the file.
[snip]
I don't want to suck the whole file into memory if I can help it, and I
can't help thinking that doing a
for line in file(logfile)
nr += 1

to count the number of lines, then reading the file again, discarding
the first nr - n records, writing the rest to a temp file, and then
renaming the files is the most efficient way to go.


Are you in a Unix environment here? If so, why not let the 'tail'
command do the work for you? Something like this:

# Close any open file objects pointing to the log file
long_log_filename = '/var/log/lotsoflines.log'
short_log_filename = '/var/log/lotsoflines.log.truncated'
lines_to_keep = 250
cmd = 'tail -%(lines)d %(long)s > %(short)s && mv %(short)s %(long)s' % {
'lines': lines_to_keep,
'short': short_log_filename,
'long': long_log_filename,
}
os.system(cmd)
# Now reopen your file objects

Or if you really want to do it in pure Python, then have a look at the
source for 'tail' and see how it finds the last N lines. In most of my
experiments, I've found that 'tail' grabs the last 10 lines out of a
1-gigabyte log file within milliseconds, while 'wc -l' takes a lot
longer to count the whole file. That's not very scientific evidence, of
course, but why don't you try it and see for yourself?

--
Robin Munn
rm***@pobox.com
Jul 18 '05 #6
Robin Munn wrote:
Or if you really want to do it in pure Python, then have a look at the
source for 'tail' and see how it finds the last N lines. In most of my
experiments, I've found that 'tail' grabs the last 10 lines out of a
1-gigabyte log file within milliseconds, while 'wc -l' takes a lot
longer to count the whole file. That's not very scientific evidence, of
course, but why don't you try it and see for yourself?


Maybe it .seeks() to .getsize()?

Gerrit.

--
258. If any one hire an ox-driver, he shall pay him six gur of corn per
year.
-- 1780 BC, Hammurabi, Code of Law
--
Asperger's Syndrome - a personal approach:
http://people.nl.linux.org/~gerrit/english/

Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Ralph Freshour | last post by:
Is there anyway to crop .jpg images so that they are all of the same width and height? I use the following math to calculate the correct aspect ratio: width / height == aspect ratio so if I...
1
by: Duncan Smith | last post by:
Hello, Can anyone advise me how to crop an EPS image generated by dislin. I can't figure out how to crop the image before generating it. Using PIL to crop the generated image I get an IOError...
13
by: Jon Yeager | last post by:
I need to display a bunch of pictures that are all of various dimensions into a fixed dimension space, like MSN Messenger does with its user photos in the chat windows. Forcing image dimensions...
3
by: Paul E Collins | last post by:
I need to load a bitmap image, crop it, and save it. By cropping, I don't mean resizing - I mean reducing it to a fixed size area. Image img = Bitmap.FromFile("in.bmp"); // what goes here?...
0
by: AmerS | last post by:
hi, I have been trying to generate image patches for a training set but have not succeded yet. The only method i tried is cropping areas of the image and store the cropped images or patches as an...
2
by: kumari | last post by:
Hi, I am facing a problem with generation of thumbnail images in php. The requirement is as follows: the client would provide the image url, description, and content through a page. and the...
0
by: lalithabhamidi | last post by:
Hai i am using GDIplus and working on VC++ 6.0.Please can u help me on Removing red eye and cropping an Image.I immediately need ur help. Thanks in advance for ur help.
1
by: jpuopolo | last post by:
All: I looked around and could not find a simple yet complete example of cropping an image in C#. I had the problem of needing to shave off the bottom-most 18 lines of a directory full of...
3
by: mcfly1204 | last post by:
I have a photo cropping tool in place that fits my needs, with one exception. When a user uploads a high res. photo, it does not fit on the screen. I have a zoom feature that allows the image to...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.