out of memory in Perl Builder

anklos · #1 October 23rd, 2008, 04:53 PM

Hi, I am writing a word stemming program, the program reads from a *.trec file to get query words in <title></title> tag, then stems the query words and output the result to another *.trec file. But perl builder promts "out of memory":

Expand|Select|Wrap|Line Numbers

 
use stem;

use WordNet::QueryData;
 
$wn = WordNet::QueryData->new;

$stemmer = stem->new($wn);
 
my $topic= "D:\\Program Files\\Perl Builder2\\MYPERL\\TrainingTopics.trec";

my $mytopic="D:\\Program Files\\Perl Builder2\\MYPERL\\myTopics.trec";

open FILE,"$topic" or die $!;

open MYTOPIC,">$mytopic" or die $!;
 
while(my $file=<FILE>)

{

    if($file=~/<title>(.+?)<\/title>/)

    {

        print MYTOPIC &stemming($file);
 
    }

    else

    {

       print MYTOPIC $file;

    }                              

}
 
sub stemming

{

     my $in = shift;

    my $out = '';
 
    foreach my $token (split /[ ,.:;?!]/, $in)

    {
 
          my @stems = $stemmer->stemWord($token);
 
             $out .= $stems[0] . ' ';
 
    }

     return $out;

}

close FILE;

close MYTOPIC;

Thanks a lot!

numberwhun · #2 October 23rd, 2008, 05:06 PM

Well, you aren't slurping the file into an array so the size of the file shouldn't make a difference so much.

It makes me wonder how much system memory you have and how many programs are running on your machine. If that it not it, then you will probably need to take this to the PerlBuilder forums as it would be outside of the realms of Perl coding.

Regards,

Jeff

**Note: are you trying to run this from Perl Builder? If so, why not run the script from the command line instead? I personally never trust a program to run my Perl script. I would rather the interpreter do it directly, without a middle man so to speak. As you can see, they always seem to find a way to cause issues.

Regards,

Jeff

KevinADC · #3 October 23rd, 2008, 06:36 PM

Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

That script looks like it would use very little memory so I don't know why its running out of memeory unless that module you are using has some known issues. WordNet::QueryData queries a database, that is possibly where the memory problem is happening. Not sure what to suggest though.

numberwhun · #4 October 23rd, 2008, 06:55 PM

Quote:

Originally Posted by KevinADC

Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

Oh, ok. Sorry, wasn't aware. I tend to hand code everything and run from the command line. If I do use an IDE, I use either Eclipse (with EPIC plugin for Perl) or Komodo. Both are awesome.

Well, now I know. :)

Ganon11 · #5 October 23rd, 2008, 07:07 PM

Quote:

Originally Posted by numberwhun

Oh, ok. Sorry, wasn't aware. I tend to hand code everything and run from the command line. If I do use an IDE, I use either Eclipse (with EPIC plugin for Perl) or Komodo. Both are awesome.

Well, now I know. :)

I'm with you, Numberwhun. Notepad++ to edit, cmd.exe to run.

KevinADC · #6 October 23rd, 2008, 07:27 PM

Writing simple scripts in a text editor is fine, but using an IDE is essential to any complex code and even not so complex. You can run syntax checks, pipe data in/out of a script (command line or CGI), write and test code on the fly, use the perl debugger in the IDE, step over code, step into code, run timers , pause a script, see and jump to any subroutine in a single glance (may vary IDE to IDE) see all the variables used in a script (may vary IDE to IDE), and much more.

With the IDE you can open multiple scripts and select the one to run and check the output too, all on the fly if you wanted to.

Make your life easier and use an IDE.

In Perl Builder all the builtin perl functions display a basic syntax usage next to the function if you want it to. So you see something like:

split

(this displays in a little window next to the function)
split /PATTERN/,EXPR,LIMIT
split /PATTERN/,EXPR
split /PATTERN/

No need to look at the documentation if you foget the simple usage syntax. That right there is such a big time saver. And if you wanted to read the entire documentation you just click on the little window and a bigger window pops up with the full perl manpage for the function.

numberwhun · #7 October 24th, 2008, 12:11 AM

Wow, didn't mean to kick you up onto your soap box (j/k). I never said I didn't use an IDE. Komodo has always been my editor of choice but have been playing more with Epic (Eclipse) due to its interesting regex tester.

I agree with you that an IDE is a very useful part of Perl coding, but its not something you cannot live without. The software I support at work is all supported without an IDE, using GVim alone. Some of the files that we make additions to are several thousand lines or more in length.

Regards,

Jeff

Icecrack · #8 October 24th, 2008, 12:53 AM

i use Crimson Editor for my Perl Coding + other Languages. (FREE WARE)
i did use Perl Builder

KevinADC · #9 October 24th, 2008, 01:05 AM

Did I forget to mention I work for PerlBuilder? J/K..... I just had too much coffee this morning.

Ganon11 · #10 October 24th, 2008, 01:29 AM

Quote:

Originally Posted by KevinADC

Did I forget to mention I work for PerlBuilder? J/K..... I just had too much coffee this morning.

*Dare I ask?* What IDE do you suggest? Is PerlBuilder free?

anklos · #11 October 24th, 2008, 01:58 AM

Quote:

Originally Posted by KevinADC

Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

That script looks like it would use very little memory so I don't know why its running out of memeory unless that module you are using has some known issues. WordNet::QueryData queries a database, that is possibly where the memory problem is happening. Not sure what to suggest though.

It has no problem when I just convert one word to its stemming words, but when I try to stemming a number of words from a file, the "out of memory" occurs. I think its probably because the wordnet's database 2. I dont know much about how the database works, hope anyone can give me more suggestion.

PS: You guys are friendly and funny, thank you!

anklos · #12 October 24th, 2008, 02:29 AM

Since there is no WordNet 3.0 for windows, I download the WordNet 2.1 and change its name to WordNet 3.0... As I said, there was no problem to convert one word to its semming word.

anklos · #13 October 24th, 2008, 03:14 AM

I find a problem in my program.

Expand|Select|Wrap|Line Numbers

if($file=~/<title>(.+?)<\/title>/)

I thinks this sentence shoud pick the content between the tag <title></title>,
not containing the tag itself. But I test my program and I find the tag is contained in my first split word. For example, <title>perl bulider</title>, the first word is "<title>perl" not "perl"

KevinADC · #14 October 24th, 2008, 03:31 AM

The regexp is probably OK:

Expand|Select|Wrap|Line Numbers

 
my $file = "test <title>Perl Builder<\/title> test";

if ($file =~ /<title>(.+?)<\/title>/) {

   print $1;

}

I think the problem is here:

Expand|Select|Wrap|Line Numbers

foreach my $token (split /[ ,.:;?!]/, $in)

There is nothing in the above regexp (split) to split the string into tokens except a space. So it gets split into "<title>Perl" and "Builder<\/title>".

anklos · #15 October 24th, 2008, 05:08 AM

Quote:

Originally Posted by KevinADC

The regexp is probably OK:

Expand|Select|Wrap|Line Numbers

 
my $file = "test <title>Perl Builder<\/title> test";

if ($file =~ /<title>(.+?)<\/title>/) {

   print $1;

}

I think the problem is here:

Expand|Select|Wrap|Line Numbers

foreach my $token (split /[ ,.:;?!]/, $in)

There is nothing in the above regexp (split) to split the string into tokens except a space. So it gets split into "<title>Perl" and "Builder<\/title>".

I see your point, you are right! Thank you very much!!

KevinADC · #16 October 24th, 2008, 06:24 AM

Quote:

Originally Posted by anklos

I see your point, you are right! Thank you very much!!

You're welcome

out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

re: out of memory in Perl Builder

/bytes/development

/bytes/it

/bytes/about