470,580 Members | 2,176 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,580 developers. It's quick & easy.

log parsing: gaps between transactions

I have a log file in this format:

YYYYMMDD HHMMSS DATADATADATADATADATA RESULTCODE

and I want to be able to parse the log and gleen information about gaps in
time between transactions. For example, it would be nice to know that 8
times in the past 24 hours we had periods over 5 minutes where no
transactions were successful. In order to do that, I need to compare each
line against the next based on result code and time elapsed. To make this
a bit more tricky, not every line in the log is in this format (approx 45%
are) but I only want to parse the lines that are. To add another problem,
I am an inexperienced perl programmer. If someone can point me in the
right direction it would be most appreciated. I have all of the O'Reilly
books and have referenced them but nothing seems to be close to what I
need. I also did an extensive search on deja before posting.

Thanks,

Arthur
Jul 19 '05 #1
1 2326
In article <qn****************************@40tude.net>, Arthur Dent
<ad***@fatadmin.com> wrote:
I have a log file in this format:

YYYYMMDD HHMMSS DATADATADATADATADATA RESULTCODE

and I want to be able to parse the log and gleen information about gaps in
time between transactions. For example, it would be nice to know that 8
times in the past 24 hours we had periods over 5 minutes where no
transactions were successful. In order to do that, I need to compare each
line against the next based on result code and time elapsed. To make this
a bit more tricky, not every line in the log is in this format (approx 45%
are) but I only want to parse the lines that are. To add another problem,
I am an inexperienced perl programmer. If someone can point me in the
right direction it would be most appreciated. I have all of the O'Reilly
books and have referenced them but nothing seems to be close to what I
need. I also did an extensive search on deja before posting.

Thanks,

Arthur


Have you tried any code? Do you have access to the perl online
documentation (try "perldoc perl" at a command line)? Check out
'perldoc perlquickre' for regular expressions and 'perldoc perlfaq5'
for file I/O.

Do you know how to read text files? You can identify the transaction
lines and extract the time information with a regular expression such
as

/(\d{8})\s+(\d\d)(\d\d)(\d\d)/

This looks for and saves an 8-digit date, followed by some white space,
followed by 3 sets of 2-digit values. If your times are in 24-hour
format, you can compute a time-since-midnight easily. Here is a sample
program:

#!/opt/perl/bin/perl

use strict;
use warnings;

my $prev = 86400;
while(<DATA>) {
if( /(\d{8})\s+(\d\d)(\d\d)(\d\d)/ ) {
my $date = $1;
my $time = $4 + 60*( $3 + 60*$2);
my $gap = $time - $prev;
if( $gap > 300 ) {
print "date = $date, time = $time, prev = $prev, gap = $gap
seconds\n";
}
$prev = $time;
}
}

__DATA__
line 1
line 2
20031127 123456 data1 code1
line 3
line 4
20031127 123457 data2 code2
line 5
line 6
20031127 123958 data3 code3
line 7
line 8
20031127 124000 data4 code4
line 9
line 10
line 11

__END__
Output:
date = 20031127, time = 45598, prev = 45297, gap = 301 seconds

You will have to modify this program to handle the case around
midnight, but that shouldn't be too hard. If you need more accurate,
absolute time, check out the Time::ParseDate module from
http://www.cpan.org/

FYI: this newsgroup is defunct. Try comp.lang.perl.misc in the future,
paying attention to the guidelines for that group, available at
http://mail.augustmail.com/~tadmc/cl...uidelines.html
Jul 19 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by Iain Mcleod | last post: by
3 posts views Thread by Robert McGregor | last post: by
11 posts views Thread by Mike P | last post: by
7 posts views Thread by Lucas Tam | last post: by
12 posts views Thread by Rami | last post: by
4 posts views Thread by ykhamitkar | last post: by
1 post views Thread by livre | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.