I have written a perl script to parse some large flat-text logs. The
logs are bzipped and come in through a pipe to STDIN. The script then
performs some regular expressions to the incoming data then prints to
STDOUT.
The script works great, but the issue I have is that the script uses
as much memory as the data coming into it. Therefore if I pipe a 800MB
file into it, the memory usage grows and grows until it reaches
approximatly 800MB, and it doesn't appear to release any of the memory
until it is completely finished. Is there a way to have Perl not use
so much of the system memory? Here is a sample of what I am doing.
#!/usr/bin/perl -w
use strict;
if ($ARGV[0]){
open STDIN, "bzcat $ARGV[0]|" or die "Cant uncompress file as a
pipe\n$!\n";
}
foreach (<STDIN>) {
chomp;
if ($_ =~ /(somedata)/) {
print "$1\n";
}
} 3 6770
James B. wrote: The script works great, but the issue I have is that the script uses as much memory as the data coming into it. Therefore if I pipe a 800MB file into it, the memory usage grows and grows until it reaches approximatly 800MB, and it doesn't appear to release any of the memory until it is completely finished. Is there a way to have Perl not use so much of the system memory? Here is a sample of what I am doing.
[...] foreach (<STDIN>) {
Well, here you are computing the full array of all lines from STDIN, then
foreach loops through them.
Why don't you use the more typical
while (<STDIN>)
which will process line by line?
jue
"James B." <ja************ @jsc.nasa.gov> wrote in message
news:75******** *************** ***@posting.goo gle.com... I have written a perl script to parse some large flat-text logs. The logs are bzipped and come in through a pipe to STDIN. The script then performs some regular expressions to the incoming data then prints to STDOUT.
The script works great, but the issue I have is that the script uses as much memory as the data coming into it. Therefore if I pipe a 800MB file into it, the memory usage grows and grows until it reaches approximatly 800MB, and it doesn't appear to release any of the memory until it is completely finished. Is there a way to have Perl not use so much of the system memory? Here is a sample of what I am doing. #!/usr/bin/perl -w
use strict;
if ($ARGV[0]){ open STDIN, "bzcat $ARGV[0]|" or die "Cant uncompress file as a pipe\n$!\n"; }
foreach (<STDIN>) { chomp; if ($_ =~ /(somedata)/) { print "$1\n"; } }
Change
foreach (<STDIN>) {
to
while (<STDIN>) {
The foreach construct reads every line in and then loops through them one at
a time.
The while construct reads one line in at a time and executes the loop.
--
Shawn
Wow, what a simple soultion. Works great, Thanks.
James This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: I Report, You Decide |
last post by:
A hosting service lungcapage has banned yabb, because it takes too much
CPU/memory, but phpBB is fine. Is that perl/php or the script itself? why
perl sucks more resource of a server than php? i thought only mod-perl will
suck not regular perl.
--
I am Social Liberal-Fiscal Conservative
Social Liberal: Forgiveness, Acceptance of Difference, Universal Healthcare,
Free College Education for Everyone
Fiscal Conservative: Small Government,...
|
by: Myron Turner |
last post by:
I'm not sure whether this question belongs entirely here or in a perl
group--but probably it requires knowledge of both.
I've written a perl module, currently in use, which does asynchronous
searches of library databases anywhere in the world. It forks off a
separate process for each database which it contacts. In the past
I've had it search successfuly through more than 1000 databases,
reporting back one one record from each.
The...
|
by: John Smith |
last post by:
Hello,
I have a rather odd question. My company is an all java/oracle shop.
We do everything is Java... no matter what it is... parsing of text
files, messaging, gui you name it. My question is this... is Perl so
much better at parsing text files and outputing that we would see a
substantial speed increase? We process about 10 million records in
flat files a day for reformatting before putting them in a DB.
Also, when it comes to...
|
by: Gary Harvey |
last post by:
I have a data intensive program that requires all data to be present in
memory. I keep running out of memory at about 2G whenever I run my program.
I tried using a 64 bit version of Perl and hit the same limit even though
the memory on the machine was 8G. Even with 32 bit addresses, I should be
able to use 4G if it is available on the machine. How can I get my perl
programs to access more than 2G of memory? Thanks.
|
by: Kirt Loki Dankmyer |
last post by:
So, I download the latest "stable" tar for perl (5.8.7) and try to compile
it on the Solaris 8 (SPARC) box that I administrate. I try all sorts of
different switches, but I can't get it to compile. I need it to be
compiled with threads.
Anyone have any wisdom on how best to do this?
Here's a transcript of my latest attempt. It's long; you might want to
skip to the bottom, where I try "make" and the fatal errors start
happening.
| |
by: MarkW |
last post by:
I hope this is the correct place to post this:
I am developing a web site for a e-commerce business I will be
running. The site I'm setting up will be 50% store, 50% content. I'm
not sure which shopping cart I'll be using yet (am looking at
os-commerce and miva) but the other part will be content. I will be
writing reviews of products, articles on the industry, and someday may
have some blogging and forums and advertising.
As for perl,...
|
by: roadbai |
last post by:
Hi all,
This is the first time to post question here, hopefully experts of perl here can give me a hand, to be honest, I am kind of new to perl, and I am struggling with the "Out of memory" issue I met when running my scripts.
Could somebody pay attention to look into the below details to give some help?
Really appreciate!!!!!
Let me give deatail info:
OS: HP 3000 MPE/iX.
|
by: surfivor |
last post by:
I may be involved in a data migration project involving databases and
creating XML feeds. Our site is PHP based, so I imagine the team might
suggest PHP, but I had a look at the PHP documentation for one of the
Pear modules for creating XML and it didn't look like much. I've used
Perl XML:Twig and I have the impression that there is more Perl stuff
available as well as the Perl stuff being well documented as I have a
Perl DBI book, a Perl...
|
by: disappearedng |
last post by:
Hi all,
I am currently planning to write my own web crawler. I know Python but
not Perl, and I am interested in knowing which of these two are a
better choice given the following scenario:
1) I/O issues: my biggest constraint in terms of resource will be
bandwidth throttle neck.
2) Efficiency issues: The crawlers have to be fast, robust and as
"memory efficient" as possible. I am running all of my crawlers on
cheap pcs with about 500...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |