469,276 Members | 1,687 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,276 developers. It's quick & easy.

perl memory usage

Hello,

This is really a reply to
http://www.thescripts.com/forum/thread50187.html

I have a script that reads in a bunch of data from txt files, then does a bunch of calculations, and outputs other txt files. I've got about 7 input txt files, each of which is 14 - 17 MB. If I open Windows Task Manager's process list I can watch Perl using increasing amounts of memory as it reads in the txt files until it chokes, at about 1.7GB (I have 2GB on my machine), which is about the 6th input txt file.

Below is a typical code block showing how I'm reading in data. Can anybody suggest a way to do the same thing without using as much memory? If not, what's the easiest way for me to tell what in my script is taking all the memory? Any help is greatly appreciated.

Expand|Select|Wrap|Line Numbers
  1. open TRAIL_FILE, "perl-inputs/fire_trails.asc.txt";
  2. my $counter = 0;
  3. while (<TRAIL_FILE>) {
  4.    chomp;
  5.    @{$trail_lines{$counter}} = split /\s+/,$_;
  6.    $counter++;
  7. }
  8. close TRAIL_FILE;
  9.  
  10. for (my $row = 6; $row < $nrows+6; $row++) {
  11.     for (my $col = 0; $col < $ncols; $col++) {
  12.         $trails{$row}{$col} = @{$trail_lines{$row}}[$col];
  13.     } # close for on $col
  14. } # close for on $row
  15.  
Feb 7 '08 #1
2 2416
Can anybody suggest a way to do the same thing without using as much memory? If not, what's the easiest way for me to tell what in my script is taking all the memory?
All your memory is being used up because you're storing all the information you're reading. What does your data look like?
Feb 7 '08 #2
KevinADC
4,059 Expert 2GB
Its the data you read into %trails (or whatever your data structure is) that is taking up memory. Maybe if you explain what you are doing someone can suggest a different way that might save memory.
Feb 7 '08 #3

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

3 posts views Thread by James B. | last post: by
8 posts views Thread by Dan | last post: by
2 posts views Thread by tomvr | last post: by
reply views Thread by cmartins | last post: by
1 post views Thread by Jean-Paul Calderone | last post: by
223 posts views Thread by Pilcrow | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.