473,323 Members | 1,622 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,323 software developers and data experts.

Using more than 2G of memory with PERL

I have a data intensive program that requires all data to be present in
memory. I keep running out of memory at about 2G whenever I run my program.
I tried using a 64 bit version of Perl and hit the same limit even though
the memory on the machine was 8G. Even with 32 bit addresses, I should be
able to use 4G if it is available on the machine. How can I get my perl
programs to access more than 2G of memory? Thanks.

Jul 19 '05 #1
2 4240
Gary Harvey wrote:
I have a data intensive program that requires all data to be present in
memory. I keep running out of memory at about 2G whenever I run my program.
I tried using a 64 bit version of Perl and hit the same limit even though
the memory on the machine was 8G. Even with 32 bit addresses, I should be
able to use 4G if it is available on the machine. How can I get my perl
programs to access more than 2G of memory? Thanks.


Can you post a snippet of code to show the circumstances
where you run out of memory - how you know that this is
what the problem is, and what you did to cause it...
Jul 19 '05 #2

"Martin Gregory" <ma************@freescale.com> wrote in message
news:cu**********@az33news01.freescale.net...
Gary Harvey wrote:
I have a data intensive program that requires all data to be present in
memory. I keep running out of memory at about 2G whenever I run my program. I tried using a 64 bit version of Perl and hit the same limit even though the memory on the machine was 8G. Even with 32 bit addresses, I should be able to use 4G if it is available on the machine. How can I get my perl
programs to access more than 2G of memory? Thanks.


Can you post a snippet of code to show the circumstances
where you run out of memory - how you know that this is
what the problem is, and what you did to cause it...

I am getting an out of memory message. I have watched the process with top
and have seen it approach 2G before running out of memory. This is a rather
long code snippet; but, where I run out of memory varies depending on the
data

---

foreach $testcase (@Test_Cases)
{
$testcase_dir = $base_dir . $forward_slash . $testcase;
if ( open(TCOVD,$testcase_dir . $forward_slash . "rd" . $forward_slash
.. "tcovd") )
{
if ($PrintProgress eq "TRUE")
{
print PROGRESS $testcase . "\n";
print $testcase . "\n";
}
@tcovd_lines = <TCOVD>;
chomp(@tcovd_lines);
close(TCOVD);
# The logic below is designed to do the following:
# If the function specified is "EvaluateUsage" then we need to
# allow for zero hit count blocks. Because of limited memory,
# we cannot keep the zero blocks for each test case. This is
# OK since every tcovd file should list all instrumented files and
# their blocks whether we hit them in a test case or not; so, we
will
# allow for zero hit count blocks for only the first test case and
only
# if "EvaluateUsage" is the specified function.
$RegExp = "";
if ($EvaluateUnreferencedBlocks eq "TRUE" && $IsFirstTestcase eq
"TRUE" )
{
$IsFirstTestcase = "FALSE";
$RegExp = $Initial_BlockLevel_RegExp;
}
else
{
$RegExp = $Subsequent_BlockLevel_RegExp;
}
for ($i = 0; $i < @tcovd_lines;)
{
# If we have encountered the beginning of source block data
# and the source block data is a ".f", ".cxx", or ".c" file;
# then, there might be block data that needs to be stored in
# the global hashe(s).
if ( $tcovd_lines[$i]=~/SRCFILE/ && ($tcovd_lines[$i]=~/\.cxx/
|| $tcovd_lines[$i]=~/\.c/ || $tcovd_lines[$i]=~/\.f/) )
{
# This confusing bit of code extracts the name of the source
library
# and the name of the file from the SRCFILE line
@temp_array_a=split(/\s+/,$tcovd_lines[$i]);
$fullpath_sourcename = $temp_array_a[1];
@temp_array_b=split(/\//,$fullpath_sourcename);
$temp_size = @temp_array_b;
$library = $temp_array_b[($temp_size -2)];
$source = $temp_array_b[($temp_size -1)];

if ($DEBUG eq "TRUE")
{
if (exists $Libraries{$library})
{
printf("%s\t%s\n",$library,$source);
}
}

$i++;
# Now that we have a library and file name the next lines
# should contain block data. We only want file and block
# data for the libraries that we are interested in
if ($AccumulateAllLibraries eq "TRUE" || exists
$Libraries{$library})
{
$this_file_has_blocks_hit = "FALSE";
$this_file_has_blocks = "FALSE";
%file_hash = ();
while ($tcovd_lines[$i]=~/^\t\t[0-9]+\t[0-9]+/)
{
if ( $tcovd_lines[$i]=~$RegExp )
{
$this_file_has_blocks_hit = "TRUE";
$this_file_has_blocks = "TRUE";
@temp_array_d=split(/\s+/,$tcovd_lines[$i]);
# OK, now we have to account for the same block
being
# listed multiple times under the same source
listing!!!
if ( exists $file_hash{$temp_array_d[1]} )
{
# Adds to the existing count
$file_hash{$temp_array_d[1]} += $temp_array_d[2];
}
else
{
# Adds new entry to hash with count
$file_hash{$temp_array_d[1]} = $temp_array_d[2];
}
}
$i++;
}
# At this point, if $this_file_has_blocks_hit is "TRUE",
# then we need to add the %file_hash to the %source_hash
# I'm leaving the ".tcov" off of the filename to save a
few
# bytes since it doesn't really add any value anyway
%temp_source_hash = ();
$block = "";
$count = "";
# Changed this test for the EvaluateUsage fuctionality.
if ( $this_file_has_blocks_hit eq "TRUE" || (
$EvaluateUnreferencedBlocks eq "TRUE" && $this_file_has_blocks eq "TRUE") )
{
$filename = $library . $forward_slash . $source;
# If this $filename already exists in $source_hash,
# then $source_hash{$filename} already has a %file_hash
# from a preceeding entry of the file. We don't want
to
# overwrite this, so we must append to it.
if ( exists $source_hash{$filename} )
{
%temp_file_hash = ();
$block = "";
$count = "";
# We already know that this $filename already has a
hash
# with some blocks in it. Now, for each of the new
blocks
# that we have collected in %file_hash, find out
which ones
# already exist in $source_hash{$filename}
foreach $block ( keys %file_hash )
{
# For existing blocks, add the new hit number to
the existing one
if ( exists $source_hash{$filename}{$block} )
{
$source_hash{$filename}{$block} +=
$file_hash{$block}
}
# Other blocks get put into $temp_hash_source to
be appended
# to $source_hash{$filename}
else
{
$temp_file_hash{$block} = $file_hash{$block};
}
}
$source_hash{$filename} = {
%temp_file_hash,%{$source_hash{$filename}} };
}
else
{
$source_hash{$filename} = { %file_hash };
}
}
} #end if (exists $Libraries{$library})

# Else, this is in a library that we don't care about so just
skip past the data
else
{
while ($tcovd_lines[$i]=~/^\t\t[0-9]+\t[0-9]+/)
{$i++;}
}

} #end if ( $tcovd_lines[$i]=~/SRCFILE/...

else
{
if ( $Debug eq "TRUE" )
{
printf("ELSE: %s\n",$tcovd_lines[$i]);
}
$i++
}

} #end for
$testcase_hash{$testcase} = {%source_hash};
%source_hash = ();
} #end if (open(TCOVD...
else
{
printf("No tcovd data for %s\n",$testcase);
}
} #end foreach $testcase

Jul 19 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: I Report, You Decide | last post by:
A hosting service lungcapage has banned yabb, because it takes too much CPU/memory, but phpBB is fine. Is that perl/php or the script itself? why perl sucks more resource of a server than php? i...
58
by: @ | last post by:
A benchmark in 2002 showed PHP is much slower in shell or when Apache has Mod_Perl. With the new PHP kissing Java's ass, Perl is once again the #1 CGI choice. Java is for a big team in short...
1
by: William H. Asquith | last post by:
I am running perl -v This is perl, v5.8.6 built for darwin-thread-multi-2level (with 2 registered patches, see perl -V for more detail) on a MacOSX 10.4, dual cpu with 2GB of RAM and 230 GB...
121
by: typingcat | last post by:
First of all, I'm an Asian and I need to input Japanese, Korean and so on. I've tried many PHP IDEs today, but almost non of them supported Unicode (UTF-8) file. I've found that the only Unicode...
19
by: Kamilche | last post by:
I have looked at many object-oriented programming frameworks out there for C. Though the ideas presented are intriguing, and I've used some of them in my own work, they all suffered some drawback...
2
by: kelly | last post by:
Hi, I don't have a code to show you, what I need are references or algorithms so that I'm in a right track. By the way, thanks for introducing me the array or arrays. Now I can continue my...
1
by: tarunjindal | last post by:
Hi I am Tarun Jindal a s/w engineer working in Satyam computers I would like to ask some question from you. We at Satyam is developing a software which will built xml for out clients. ...
21
KevinADC
by: KevinADC | last post by:
Note: You may skip to the end of the article if all you want is the perl code. Introduction Uploading files from a local computer to a remote web server has many useful purposes, the most...
12
by: pankajit09 | last post by:
The perl code is as below : use strict; use Spreadsheet::ParseExcel; my $parser = Spreadsheet::ParseExcel->new( CellHandler => \&cell_handler, NotSetCell => 1 ...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.