469,327 Members | 1,200 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,327 developers. It's quick & easy.

Parsing Multiple Log Files

hi all

i have started reading perl recently... my project is dealing with log files for which i will have to develop scripts to parse it and many more that performs many operationst. now my immediate requirement is that i should create a script thats able to accept multiple log files as input.. i am able to provide single log file as input but dont know how to proceed with multiple log files. can anyone show light in this.

i would be greatful if i get any kind of help.

thanks in prior

sunil
Mar 19 '07 #1
9 3831
@sunilmehta

You know how to process a single log file. So, keep multiple log files in an array of file names (which you, e.g. retrieve from a file). The iterate the list and process each individual log file as you do already:

Expand|Select|Wrap|Line Numbers
  1. @logFiles = ...      # retrieve list of file names from a file or whatever
  2.  
  3. for $file ( @logFiles  {
  4.   doWhatYouDoForSingleLogFile($files);
  5. }
Greetz, Doc
Mar 19 '07 #2
doc

thanks for ur guidence.i tried many ways to take in multiple files, but everything showing errors can u tell me some way how can i take in multiple files.

would be so helpful

thanks
sunil
Mar 20 '07 #3
miller
1,089 Expert 1GB
Hi Sunil,

What ways have you tried? You've already stated that you know how to parse a single log file. To parse multiple ones, you simply do it sequentially using a for loop as doc suggested.

What errors are you getting in your attempts? If you provide the code that you are working with it is more likely that we'll be able to point out exactly where you need help.

The only challenge I can forsee in what you're try to do is deciding on how you want to accumulate the list of files. You can do it from the command line, you could process a specific directory and/or tree, you could hard code the list. Which method you use is not something we can dictate on you though, it must match your requirements.

So tell us what you've tried so far and maybe we can help, as what you've attempting is not difficult.

Regards,
- Miller
Mar 20 '07 #4
miller

i just took one file and tired displaying characters and number of lines in it.... the code i have used for it is

Expand|Select|Wrap|Line Numbers
  1. use strict;
  2.  
  3. my $chars = 0;
  4. my $words = 0;
  5. my $lines = 0;
  6.  
  7. sub tokenize {
  8.     $_ = $_[0];
  9.     s/\s+/\n/g;
  10.     s/^\n//;
  11.     s/$/\n/;
  12.     s/([.,!?:;,])\n/\n$1\n/g;
  13.     s/\n(["'`])([^\n])/\n$1\n$2/g;
  14.     s/([^\n])(["'`])\n/$1\n$2\n/g;
  15.     s/([^\n])([.,])\n/$1\n$2\n/g;
  16.     s/\n([A-Z])\n\./\n$1./g;
  17.     s/\n\.\n([^"A-Z])/\.\n$1/g;
  18.     s/(\.[A-Z]+)\n\.\n/$1.\n/g;
  19.     s/([^\n])'s\n/$1\n's\n/g;
  20.     s/([^\n])n't\n/$1\nn't\n/g;
  21.     s/([^\n])'re\n/$1\n're\n/g;
  22.     s/\n\$([^\n])/\n\$\n$1/g;
  23.     s/([^\n])%\n/$1\n%\n/g;
  24.     s/Mr\n\.\n/Mr.\n/g;
  25.     return (split(/\n/, $_));
  26. }
  27.  
  28. open(INFILE,"ApacheLFAsample.log") or die("cannot open input file");
  29.  
  30. while (<INFILE>) {
  31.     $chars += length;
  32.     my @tokens = &tokenize($_);
  33.     foreach my $token (@tokens) {
  34.         if ($token =~ /[a-zA-Z]/) { $words++; }
  35.     }
  36.     $lines++;
  37. }
  38.  
  39. close(INFILE);
  40. print "Found $lines lines, $words words and $chars characters.\n";
  41.  
  42. exit(0);
  43.  
now this takes single file.... now i want to do the same for multiple files means what should i do..

i know these are basic things . since there is no one to guide me am finding it too difficult.would be greatful to whatever help i get.

thanks
sunil
Mar 21 '07 #5
@sunilmehta

Just to give you an idea:
Expand|Select|Wrap|Line Numbers
  1. @arr = (
  2.   "/this/is/one/path",
  3.   "/this/is/another/path",
  4. );
  5.  
  6. for $path ( @arr ) {
  7.   # process the single file $path here !
  8. }
Greetz, Doc
Mar 21 '07 #6
KevinADC
4,059 Expert 2GB
Perl has a built in mechanism for processing a list or array of files:


Expand|Select|Wrap|Line Numbers
  1. @ARGV = qw(ApacheLFAsample.log  error.log  blah.log);
  2.  
  3. while (<>) {
  4.    your processing code here
  5. }
Each file listed in @ARGV will be opened and read by the <> operator. You don't have to use any open() or close() functions, perl takes care of that when you do it like this. I'm pretty sure it's covered in this tutorial somewhere:

http://perldoc.perl.org/perlopentut.html
Mar 21 '07 #7
HI ALL

thanks for all ur help. but now it takes mulitple files but the result is not correctly coming

the code i have used is

# count characters, words and lines in two files

Expand|Select|Wrap|Line Numbers
  1. use strict;
  2. my $chars = 0;
  3. my $words = 0;
  4. my $lines = 0;
  5. my ($token,@tokens);
  6. my ($file,@files);
  7. sub tokenize {
  8. $_ = $_[0];
  9. s/\s+/\n/g;
  10. s/^\n//;
  11. s/$/\n/;
  12. s/([.,!?:;,])\n/\n$1\n/g;
  13. s/\n(["'`])([^\n])/\n$1\n$2/g;
  14. s/([^\n])(["'`])\n/$1\n$2\n/g;
  15. s/([^\n])([.,])\n/$1\n$2\n/g;
  16. s/\n([A-Z])\n\./\n$1./g;
  17. s/\n\.\n([^"A-Z])/\.\n$1/g;
  18. s/(\.[A-Z]+)\n\.\n/$1.\n/g;
  19. s/([^\n])'s\n/$1\n's\n/g;
  20. s/([^\n])n't\n/$1\nn't\n/g;
  21. s/([^\n])'re\n/$1\n're\n/g;
  22. s/\n\$([^\n])/\n\$\n$1/g;
  23. s/([^\n])%\n/$1\n%\n/g;
  24. s/Mr\n\.\n/Mr.\n/g;
  25. return(split(/\n/,$_));
  26. }
  27.  
  28. @ARGV = qw("ApacheLFAsample.log","apache.txt" );
  29. foreach $file (@ARGV)
  30. $chars += length;
  31. @tokens = &tokenize($_);
  32. foreach $token (@tokens) 
  33. {
  34. if ($token =~ /[a-zA-Z]/) { $words++; }
  35. }
  36. $lines++;
  37. }
  38. print "Found $lines lines, $words words and $chars characters.\n";
  39. exit(0);
The thing i want here is the csript should read both files and should tell the number of lines,chars etc corresponding to each file

the output am getting now does not even match with a single file contents

expecting some help

thanks to whatever help i get

sunil
Mar 22 '07 #8
KevinADC
4,059 Expert 2GB
like this, don't change anything in the code, don't add a foreach() loop, don't add any commas or quotes to the filenames in the @ARGV array:.

Expand|Select|Wrap|Line Numbers
  1. use strict;
  2. my $chars = 0;
  3. my $words = 0;
  4. my $lines = 0;
  5. my ($token,@tokens);
  6. my ($file,@files);
  7. sub tokenize {
  8. $_ = $_[0];
  9. s/\s+/\n/g;
  10. s/^\n//;
  11. s/$/\n/;
  12. s/([.,!?:;,])\n/\n$1\n/g;
  13. s/\n(["'`])([^\n])/\n$1\n$2/g;
  14. s/([^\n])(["'`])\n/$1\n$2\n/g;
  15. s/([^\n])([.,])\n/$1\n$2\n/g;
  16. s/\n([A-Z])\n\./\n$1./g;
  17. s/\n\.\n([^"A-Z])/\.\n$1/g;
  18. s/(\.[A-Z]+)\n\.\n/$1.\n/g;
  19. s/([^\n])'s\n/$1\n's\n/g;
  20. s/([^\n])n't\n/$1\nn't\n/g;
  21. s/([^\n])'re\n/$1\n're\n/g;
  22. s/\n\$([^\n])/\n\$\n$1/g;
  23. s/([^\n])%\n/$1\n%\n/g;
  24. s/Mr\n\.\n/Mr.\n/g;
  25. return(split(/\n/,$_));
  26. }
  27.  
  28. @ARGV = qw(ApacheLFAsample.log apache.txt);
  29. while(<>){
  30.    $chars += length;
  31.    @tokens = &tokenize($_);
  32.    foreach $token (@tokens){
  33.       if ($token =~ /[a-zA-Z]/) { 
  34.          $words++;
  35.       }
  36.    }
  37.    $lines++;
  38. }
  39. print "Found $lines lines, $words words and $chars characters.\n";
  40. exit(0);
  41.  
and please start using the code tags to post code.
Mar 22 '07 #9
thanks a lot kevin its working. will rectify my errors.
Mar 22 '07 #10

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

2 posts views Thread by Michael Hogan | last post: by
reply views Thread by rick_muller | last post: by
3 posts views Thread by David Svoboda | last post: by
1 post views Thread by Robert Neville | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by Purva khokhar | last post: by
reply views Thread by haryvincent176 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.