472,989 Members | 2,803 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,989 software developers and data experts.

Perl - Memory Issues

I have written a perl script to parse some large flat-text logs. The
logs are bzipped and come in through a pipe to STDIN. The script then
performs some regular expressions to the incoming data then prints to
STDOUT.

The script works great, but the issue I have is that the script uses
as much memory as the data coming into it. Therefore if I pipe a 800MB
file into it, the memory usage grows and grows until it reaches
approximatly 800MB, and it doesn't appear to release any of the memory
until it is completely finished. Is there a way to have Perl not use
so much of the system memory? Here is a sample of what I am doing.

#!/usr/bin/perl -w

use strict;

if ($ARGV[0]){
open STDIN, "bzcat $ARGV[0]|" or die "Cant uncompress file as a
pipe\n$!\n";
}

foreach (<STDIN>) {
chomp;
if ($_ =~ /(somedata)/) {
print "$1\n";
}
}
Jul 19 '05 #1
3 6737
James B. wrote:
The script works great, but the issue I have is that the script uses
as much memory as the data coming into it. Therefore if I pipe a 800MB
file into it, the memory usage grows and grows until it reaches
approximatly 800MB, and it doesn't appear to release any of the memory
until it is completely finished. Is there a way to have Perl not use
so much of the system memory? Here is a sample of what I am doing.
[...] foreach (<STDIN>) {


Well, here you are computing the full array of all lines from STDIN, then
foreach loops through them.
Why don't you use the more typical
while (<STDIN>)
which will process line by line?

jue
Jul 19 '05 #2
"James B." <ja************@jsc.nasa.gov> wrote in message
news:75**************************@posting.google.c om...
I have written a perl script to parse some large flat-text logs. The
logs are bzipped and come in through a pipe to STDIN. The script then
performs some regular expressions to the incoming data then prints to
STDOUT.

The script works great, but the issue I have is that the script uses
as much memory as the data coming into it. Therefore if I pipe a 800MB
file into it, the memory usage grows and grows until it reaches
approximatly 800MB, and it doesn't appear to release any of the memory
until it is completely finished. Is there a way to have Perl not use
so much of the system memory? Here is a sample of what I am doing.

#!/usr/bin/perl -w

use strict;

if ($ARGV[0]){
open STDIN, "bzcat $ARGV[0]|" or die "Cant uncompress file as a
pipe\n$!\n";
}

foreach (<STDIN>) {
chomp;
if ($_ =~ /(somedata)/) {
print "$1\n";
}
}

Change

foreach (<STDIN>) {

to

while (<STDIN>) {

The foreach construct reads every line in and then loops through them one at
a time.
The while construct reads one line in at a time and executes the loop.

--
Shawn

Jul 19 '05 #3
Wow, what a simple soultion. Works great, Thanks.

James
Jul 19 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: I Report, You Decide | last post by:
A hosting service lungcapage has banned yabb, because it takes too much CPU/memory, but phpBB is fine. Is that perl/php or the script itself? why perl sucks more resource of a server than php? i...
3
by: Myron Turner | last post by:
I'm not sure whether this question belongs entirely here or in a perl group--but probably it requires knowledge of both. I've written a perl module, currently in use, which does asynchronous...
3
by: John Smith | last post by:
Hello, I have a rather odd question. My company is an all java/oracle shop. We do everything is Java... no matter what it is... parsing of text files, messaging, gui you name it. My question...
2
by: Gary Harvey | last post by:
I have a data intensive program that requires all data to be present in memory. I keep running out of memory at about 2G whenever I run my program. I tried using a 64 bit version of Perl and hit...
0
by: Kirt Loki Dankmyer | last post by:
So, I download the latest "stable" tar for perl (5.8.7) and try to compile it on the Solaris 8 (SPARC) box that I administrate. I try all sorts of different switches, but I can't get it to compile....
3
by: MarkW | last post by:
I hope this is the correct place to post this: I am developing a web site for a e-commerce business I will be running. The site I'm setting up will be 50% store, 50% content. I'm not sure which...
1
by: roadbai | last post by:
Hi all, This is the first time to post question here, hopefully experts of perl here can give me a hand, to be honest, I am kind of new to perl, and I am struggling with the "Out of memory" issue I...
6
by: surfivor | last post by:
I may be involved in a data migration project involving databases and creating XML feeds. Our site is PHP based, so I imagine the team might suggest PHP, but I had a look at the PHP documentation...
12
by: disappearedng | last post by:
Hi all, I am currently planning to write my own web crawler. I know Python but not Perl, and I am interested in knowing which of these two are a better choice given the following scenario: 1)...
0
by: Aliciasmith | last post by:
In an age dominated by smartphones, having a mobile app for your business is no longer an option; it's a necessity. Whether you're a startup or an established enterprise, finding the right mobile app...
2
by: giovanniandrean | last post by:
The energy model is structured as follows and uses excel sheets to give input data: 1-Utility.py contains all the functions needed to calculate the variables and other minor things (mentions...
4
NeoPa
by: NeoPa | last post by:
Hello everyone. I find myself stuck trying to find the VBA way to get Access to create a PDF of the currently-selected (and open) object (Form or Report). I know it can be done by selecting :...
3
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...
1
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...
3
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
0
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...
3
SueHopson
by: SueHopson | last post by:
Hi All, I'm trying to create a single code (run off a button that calls the Private Sub) for our parts list report that will allow the user to filter by either/both PartVendor and PartType. On...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.