Parsing Multiple Log Files

[sunilmehta]

hi all

i have started reading perl recently... my project is dealing with log files for which i will have to develop scripts to parse it and many more that performs many operationst. now my immediate requirement is that i should create a script thats able to accept multiple log files as input.. i am able to provide single log file as input but dont know how to proceed with multiple log files. can anyone show light in this.

i would be greatful if i get any kind of help.

thanks in prior

sunil

[docsnyder]

@sunilmehta

You know how to process a single log file. So, keep multiple log files in an array of file names (which you, e.g. retrieve from a file). The iterate the list and process each individual log file as you do already:

@logFiles = ...      # retrieve list of file names from a file or whatever
 
for $file ( @logFiles  {
  doWhatYouDoForSingleLogFile($files);
}

Greetz, Doc

[sunilmehta]

doc

thanks for ur guidence.i tried many ways to take in multiple files, but everything showing errors can u tell me some way how can i take in multiple files.

would be so helpful

thanks
sunil

[miller]

Hi Sunil,

What ways have you tried? You've already stated that you know how to parse a single log file. To parse multiple ones, you simply do it sequentially using a for loop as doc suggested.

What errors are you getting in your attempts? If you provide the code that you are working with it is more likely that we'll be able to point out exactly where you need help.

The only challenge I can forsee in what you're try to do is deciding on how you want to accumulate the list of files. You can do it from the command line, you could process a specific directory and/or tree, you could hard code the list. Which method you use is not something we can dictate on you though, it must match your requirements.

So tell us what you've tried so far and maybe we can help, as what you've attempting is not difficult.

Regards,
- Miller

[sunilmehta]

miller

i just took one file and tired displaying characters and number of lines in it.... the code i have used for it is

use strict;
 
my $chars = 0;
my $words = 0;
my $lines = 0;
 
sub tokenize {
    $_ = $_[0];
    s/\s+/\n/g;
    s/^\n//;
    s/$/\n/;
    s/([.,!?:;,])\n/\n$1\n/g;
    s/\n(["'`])([^\n])/\n$1\n$2/g;
    s/([^\n])(["'`])\n/$1\n$2\n/g;
    s/([^\n])([.,])\n/$1\n$2\n/g;
    s/\n([A-Z])\n\./\n$1./g;
    s/\n\.\n([^"A-Z])/\.\n$1/g;
    s/(\.[A-Z]+)\n\.\n/$1.\n/g;
    s/([^\n])'s\n/$1\n's\n/g;
    s/([^\n])n't\n/$1\nn't\n/g;
    s/([^\n])'re\n/$1\n're\n/g;
    s/\n\$([^\n])/\n\$\n$1/g;
    s/([^\n])%\n/$1\n%\n/g;
    s/Mr\n\.\n/Mr.\n/g;
    return (split(/\n/, $_));
}
 
open(INFILE,"ApacheLFAsample.log") or die("cannot open input file");
 
while (<INFILE>) {
    $chars += length;
    my @tokens = &tokenize($_);
    foreach my $token (@tokens) {
        if ($token =~ /[a-zA-Z]/) { $words++; }
    }
    $lines++;
}
 
close(INFILE);
print "Found $lines lines, $words words and $chars characters.\n";
 
exit(0);
 

now this takes single file.... now i want to do the same for multiple files means what should i do..

i know these are basic things . since there is no one to guide me am finding it too difficult.would be greatful to whatever help i get.

thanks
sunil

[docsnyder]

@sunilmehta

Just to give you an idea:

@arr = (
  "/this/is/one/path",
  "/this/is/another/path",
);
 
for $path ( @arr ) {
  # process the single file $path here !
}

Greetz, Doc

[KevinADC]

Perl has a built in mechanism for processing a list or array of files:

@ARGV = qw(ApacheLFAsample.log  error.log  blah.log);
 
while (<>) {
   your processing code here
}

Each file listed in @ARGV will be opened and read by the <> operator. You don't have to use any open() or close() functions, perl takes care of that when you do it like this. I'm pretty sure it's covered in this tutorial somewhere:

http://perldoc.perl.org/perlopentut.html

[sunilmehta]

HI ALL

thanks for all ur help. but now it takes mulitple files but the result is not correctly coming

the code i have used is

# count characters, words and lines in two files

use strict;
my $chars = 0;
my $words = 0;
my $lines = 0;
my ($token,@tokens);
my ($file,@files);
sub tokenize {
$_ = $_[0];
s/\s+/\n/g;
s/^\n//;
s/$/\n/;
s/([.,!?:;,])\n/\n$1\n/g;
s/\n(["'`])([^\n])/\n$1\n$2/g;
s/([^\n])(["'`])\n/$1\n$2\n/g;
s/([^\n])([.,])\n/$1\n$2\n/g;
s/\n([A-Z])\n\./\n$1./g;
s/\n\.\n([^"A-Z])/\.\n$1/g;
s/(\.[A-Z]+)\n\.\n/$1.\n/g;
s/([^\n])'s\n/$1\n's\n/g;
s/([^\n])n't\n/$1\nn't\n/g;
s/([^\n])'re\n/$1\n're\n/g;
s/\n\$([^\n])/\n\$\n$1/g;
s/([^\n])%\n/$1\n%\n/g;
s/Mr\n\.\n/Mr.\n/g;
return(split(/\n/,$_));
}
 
@ARGV = qw("ApacheLFAsample.log","apache.txt" );
foreach $file (@ARGV)
{ 
$chars += length;
@tokens = &tokenize($_);
foreach $token (@tokens) 
{
if ($token =~ /[a-zA-Z]/) { $words++; }
}
$lines++;
}
print "Found $lines lines, $words words and $chars characters.\n";
exit(0);

The thing i want here is the csript should read both files and should tell the number of lines,chars etc corresponding to each file

the output am getting now does not even match with a single file contents

expecting some help

thanks to whatever help i get

sunil

[KevinADC]

like this, don't change anything in the code, don't add a foreach() loop, don't add any commas or quotes to the filenames in the @ARGV array:.

use strict;
my $chars = 0;
my $words = 0;
my $lines = 0;
my ($token,@tokens);
my ($file,@files);
sub tokenize {
$_ = $_[0];
s/\s+/\n/g;
s/^\n//;
s/$/\n/;
s/([.,!?:;,])\n/\n$1\n/g;
s/\n(["'`])([^\n])/\n$1\n$2/g;
s/([^\n])(["'`])\n/$1\n$2\n/g;
s/([^\n])([.,])\n/$1\n$2\n/g;
s/\n([A-Z])\n\./\n$1./g;
s/\n\.\n([^"A-Z])/\.\n$1/g;
s/(\.[A-Z]+)\n\.\n/$1.\n/g;
s/([^\n])'s\n/$1\n's\n/g;
s/([^\n])n't\n/$1\nn't\n/g;
s/([^\n])'re\n/$1\n're\n/g;
s/\n\$([^\n])/\n\$\n$1/g;
s/([^\n])%\n/$1\n%\n/g;
s/Mr\n\.\n/Mr.\n/g;
return(split(/\n/,$_));
}
 
@ARGV = qw(ApacheLFAsample.log apache.txt);
while(<>){
   $chars += length;
   @tokens = &tokenize($_);
   foreach $token (@tokens){
      if ($token =~ /[a-zA-Z]/) { 
         $words++;
      }
   }
   $lines++;
}
print "Found $lines lines, $words words and $chars characters.\n";
exit(0);
 

and please start using the code tags to post code.

[sunilmehta]

thanks a lot kevin its working. will rectify my errors.