473,692 Members | 2,525 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Parsing Multiple Log Files

10 New Member
hi all

i have started reading perl recently... my project is dealing with log files for which i will have to develop scripts to parse it and many more that performs many operationst. now my immediate requirement is that i should create a script thats able to accept multiple log files as input.. i am able to provide single log file as input but dont know how to proceed with multiple log files. can anyone show light in this.

i would be greatful if i get any kind of help.

thanks in prior

sunil
Mar 19 '07 #1
9 4141
docsnyder
88 New Member
@sunilmehta

You know how to process a single log file. So, keep multiple log files in an array of file names (which you, e.g. retrieve from a file). The iterate the list and process each individual log file as you do already:

Expand|Select|Wrap|Line Numbers
  1. @logFiles = ...      # retrieve list of file names from a file or whatever
  2.  
  3. for $file ( @logFiles  {
  4.   doWhatYouDoForSingleLogFile($files);
  5. }
Greetz, Doc
Mar 19 '07 #2
sunilmehta
10 New Member
doc

thanks for ur guidence.i tried many ways to take in multiple files, but everything showing errors can u tell me some way how can i take in multiple files.

would be so helpful

thanks
sunil
Mar 20 '07 #3
miller
1,089 Recognized Expert Top Contributor
Hi Sunil,

What ways have you tried? You've already stated that you know how to parse a single log file. To parse multiple ones, you simply do it sequentially using a for loop as doc suggested.

What errors are you getting in your attempts? If you provide the code that you are working with it is more likely that we'll be able to point out exactly where you need help.

The only challenge I can forsee in what you're try to do is deciding on how you want to accumulate the list of files. You can do it from the command line, you could process a specific directory and/or tree, you could hard code the list. Which method you use is not something we can dictate on you though, it must match your requirements.

So tell us what you've tried so far and maybe we can help, as what you've attempting is not difficult.

Regards,
- Miller
Mar 20 '07 #4
sunilmehta
10 New Member
miller

i just took one file and tired displaying characters and number of lines in it.... the code i have used for it is

Expand|Select|Wrap|Line Numbers
  1. use strict;
  2.  
  3. my $chars = 0;
  4. my $words = 0;
  5. my $lines = 0;
  6.  
  7. sub tokenize {
  8.     $_ = $_[0];
  9.     s/\s+/\n/g;
  10.     s/^\n//;
  11.     s/$/\n/;
  12.     s/([.,!?:;,])\n/\n$1\n/g;
  13.     s/\n(["'`])([^\n])/\n$1\n$2/g;
  14.     s/([^\n])(["'`])\n/$1\n$2\n/g;
  15.     s/([^\n])([.,])\n/$1\n$2\n/g;
  16.     s/\n([A-Z])\n\./\n$1./g;
  17.     s/\n\.\n([^"A-Z])/\.\n$1/g;
  18.     s/(\.[A-Z]+)\n\.\n/$1.\n/g;
  19.     s/([^\n])'s\n/$1\n's\n/g;
  20.     s/([^\n])n't\n/$1\nn't\n/g;
  21.     s/([^\n])'re\n/$1\n're\n/g;
  22.     s/\n\$([^\n])/\n\$\n$1/g;
  23.     s/([^\n])%\n/$1\n%\n/g;
  24.     s/Mr\n\.\n/Mr.\n/g;
  25.     return (split(/\n/, $_));
  26. }
  27.  
  28. open(INFILE,"ApacheLFAsample.log") or die("cannot open input file");
  29.  
  30. while (<INFILE>) {
  31.     $chars += length;
  32.     my @tokens = &tokenize($_);
  33.     foreach my $token (@tokens) {
  34.         if ($token =~ /[a-zA-Z]/) { $words++; }
  35.     }
  36.     $lines++;
  37. }
  38.  
  39. close(INFILE);
  40. print "Found $lines lines, $words words and $chars characters.\n";
  41.  
  42. exit(0);
  43.  
now this takes single file.... now i want to do the same for multiple files means what should i do..

i know these are basic things . since there is no one to guide me am finding it too difficult.would be greatful to whatever help i get.

thanks
sunil
Mar 21 '07 #5
docsnyder
88 New Member
@sunilmehta

Just to give you an idea:
Expand|Select|Wrap|Line Numbers
  1. @arr = (
  2.   "/this/is/one/path",
  3.   "/this/is/another/path",
  4. );
  5.  
  6. for $path ( @arr ) {
  7.   # process the single file $path here !
  8. }
Greetz, Doc
Mar 21 '07 #6
KevinADC
4,059 Recognized Expert Specialist
Perl has a built in mechanism for processing a list or array of files:


Expand|Select|Wrap|Line Numbers
  1. @ARGV = qw(ApacheLFAsample.log  error.log  blah.log);
  2.  
  3. while (<>) {
  4.    your processing code here
  5. }
Each file listed in @ARGV will be opened and read by the <> operator. You don't have to use any open() or close() functions, perl takes care of that when you do it like this. I'm pretty sure it's covered in this tutorial somewhere:

http://perldoc.perl.org/perlopentut.html
Mar 21 '07 #7
sunilmehta
10 New Member
HI ALL

thanks for all ur help. but now it takes mulitple files but the result is not correctly coming

the code i have used is

# count characters, words and lines in two files

Expand|Select|Wrap|Line Numbers
  1. use strict;
  2. my $chars = 0;
  3. my $words = 0;
  4. my $lines = 0;
  5. my ($token,@tokens);
  6. my ($file,@files);
  7. sub tokenize {
  8. $_ = $_[0];
  9. s/\s+/\n/g;
  10. s/^\n//;
  11. s/$/\n/;
  12. s/([.,!?:;,])\n/\n$1\n/g;
  13. s/\n(["'`])([^\n])/\n$1\n$2/g;
  14. s/([^\n])(["'`])\n/$1\n$2\n/g;
  15. s/([^\n])([.,])\n/$1\n$2\n/g;
  16. s/\n([A-Z])\n\./\n$1./g;
  17. s/\n\.\n([^"A-Z])/\.\n$1/g;
  18. s/(\.[A-Z]+)\n\.\n/$1.\n/g;
  19. s/([^\n])'s\n/$1\n's\n/g;
  20. s/([^\n])n't\n/$1\nn't\n/g;
  21. s/([^\n])'re\n/$1\n're\n/g;
  22. s/\n\$([^\n])/\n\$\n$1/g;
  23. s/([^\n])%\n/$1\n%\n/g;
  24. s/Mr\n\.\n/Mr.\n/g;
  25. return(split(/\n/,$_));
  26. }
  27.  
  28. @ARGV = qw("ApacheLFAsample.log","apache.txt" );
  29. foreach $file (@ARGV)
  30. $chars += length;
  31. @tokens = &tokenize($_);
  32. foreach $token (@tokens) 
  33. {
  34. if ($token =~ /[a-zA-Z]/) { $words++; }
  35. }
  36. $lines++;
  37. }
  38. print "Found $lines lines, $words words and $chars characters.\n";
  39. exit(0);
The thing i want here is the csript should read both files and should tell the number of lines,chars etc corresponding to each file

the output am getting now does not even match with a single file contents

expecting some help

thanks to whatever help i get

sunil
Mar 22 '07 #8
KevinADC
4,059 Recognized Expert Specialist
like this, don't change anything in the code, don't add a foreach() loop, don't add any commas or quotes to the filenames in the @ARGV array:.

Expand|Select|Wrap|Line Numbers
  1. use strict;
  2. my $chars = 0;
  3. my $words = 0;
  4. my $lines = 0;
  5. my ($token,@tokens);
  6. my ($file,@files);
  7. sub tokenize {
  8. $_ = $_[0];
  9. s/\s+/\n/g;
  10. s/^\n//;
  11. s/$/\n/;
  12. s/([.,!?:;,])\n/\n$1\n/g;
  13. s/\n(["'`])([^\n])/\n$1\n$2/g;
  14. s/([^\n])(["'`])\n/$1\n$2\n/g;
  15. s/([^\n])([.,])\n/$1\n$2\n/g;
  16. s/\n([A-Z])\n\./\n$1./g;
  17. s/\n\.\n([^"A-Z])/\.\n$1/g;
  18. s/(\.[A-Z]+)\n\.\n/$1.\n/g;
  19. s/([^\n])'s\n/$1\n's\n/g;
  20. s/([^\n])n't\n/$1\nn't\n/g;
  21. s/([^\n])'re\n/$1\n're\n/g;
  22. s/\n\$([^\n])/\n\$\n$1/g;
  23. s/([^\n])%\n/$1\n%\n/g;
  24. s/Mr\n\.\n/Mr.\n/g;
  25. return(split(/\n/,$_));
  26. }
  27.  
  28. @ARGV = qw(ApacheLFAsample.log apache.txt);
  29. while(<>){
  30.    $chars += length;
  31.    @tokens = &tokenize($_);
  32.    foreach $token (@tokens){
  33.       if ($token =~ /[a-zA-Z]/) { 
  34.          $words++;
  35.       }
  36.    }
  37.    $lines++;
  38. }
  39. print "Found $lines lines, $words words and $chars characters.\n";
  40. exit(0);
  41.  
and please start using the code tags to post code.
Mar 22 '07 #9
sunilmehta
10 New Member
thanks a lot kevin its working. will rectify my errors.
Mar 22 '07 #10

Sign in to post your reply or Sign up for a free account.

Similar topics

2
3901
by: Michael Hogan | last post by:
I want to pars a playlist file for three different varibles, so I can save them as mp3 files. I am using: strTEMPURL = GetUrlSource(Text1.Text) to put the entire .pls file into a strTEMPURL varible. I want to grab .pls files from shoutcast and break them into varibles. ----------------Begin winamp.pls file-------------------
4
3823
by: Fuzzyman | last post by:
There have been a couple of config file 'systems' announced recently, that focus on building more powerful and complex configuration files. ConfigObj is a module to enable you to much more *simply* access config files. This is version 3, which is a big overhaul. It extends ConfigObj to reading config files with sections and various other simplifications. I find ConfigObj extremely easy to use and use it for reading config files and data...
0
2305
by: rick_muller | last post by:
I'm interested in parsing a (simple) Makefile using Python. I need to write a packager for a program I'm supporting, and would like to read the list of files in the makefile so that I only have to update that list once. That is, I have a statement (actually multiple statements) in the Makefile that looks like: FILES = a.f b.f c.c d.py e.c \ f.f g.f h.c
35
3311
by: .:mmac:. | last post by:
I have a bunch of files (Playlist files for media player) and I am trying to create an automatically generated web page that includes the last 20 or 30 of these files. The files are created every week and are named XX-XX-XX.ASX where the X's represent the date i.e. 05-22-05.asx The files are a specific format and will always contain tags like the following: <TITLE>My media file title</TITLE> <AUTHOR>Media file author</AUTHOR> <Ref href =...
3
1602
by: David Svoboda | last post by:
I have a server program that takes commands and acts on them. The server program can also take these commands from an input file or standard input (mainly for testing purposes). As such, I often have files full of input commands to feed to the server. Right now the commands that the server takes are well-defined, but not in XML. Since the commands are not self-delimiting, I have to prepend each command with a 'length' number indicating...
2
1877
by: Adrian | last post by:
Hi All, Is there anyway to change what isspace thinks is a space character. I am parsing some log files and it would be nice to just read a field as what ever is between quotes or between 's ie clf log files I know I can go char by char or find_last_of etc, but I would like to know if it is possible with locales and facets? Also are there any suggestions for outputting in multiple languages? Can I use locales again? I was just...
1
1763
by: Robert Neville | last post by:
Basically, I want to create a table in html, xml, or xslt; with any number of regular expressions; a script (Perl or Python) which reads each table row (regex and replacement); and performs the replacement on any file name, folder, or text file (e.g. css, php, html). For example, I often rename my mp3 (files); the folder holding the mp3 files; and replace these renamed values in a playlist/m3u/xml file. The table should hold clean...
2
1449
by: drjay | last post by:
I'm trying to get my script to parse a bunch of files and grab data between the <title></> and <blah></> tags. Yes yes, I'm parsing html with regex, it works though. :) The issue I have is sometimes there is one line, sometimes 30 lines, between <title> and <blah> so I can't just .+ it all the way. Plus there are multiple <blah> tags in each file. I'm looking for a way for to scan the file for <title>, assign to $1, then search for every...
3
4508
by: GazK | last post by:
I have been using an xml parsing script to parse a number of rss feeds and return relevant results to a database. The script has worked well for a couple of years, despite having very crude error-trapping (if it finds an error in one of the xml files, the script stops). Recently, the script has stopped working because one of the xml files is badly formed. So I decided to rewrite the script with better error trapping; the script should...
0
8611
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8547
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8970
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8810
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7639
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6462
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5822
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4329
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
2984
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.