By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
437,776 Members | 1,777 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 437,776 IT Pros & Developers. It's quick & easy.

out of memory in Perl Builder

P: 30
Hi, I am writing a word stemming program, the program reads from a *.trec file to get query words in <title></title> tag, then stems the query words and output the result to another *.trec file. But perl builder promts "out of memory":
Expand|Select|Wrap|Line Numbers
  1. use stem;
  2. use WordNet::QueryData;
  3.  
  4. $wn = WordNet::QueryData->new;
  5. $stemmer = stem->new($wn);
  6.  
  7.  
  8. my $topic= "D:\\Program Files\\Perl Builder2\\MYPERL\\TrainingTopics.trec";
  9. my $mytopic="D:\\Program Files\\Perl Builder2\\MYPERL\\myTopics.trec";
  10. open FILE,"$topic" or die $!;
  11. open MYTOPIC,">$mytopic" or die $!;
  12.  
  13. while(my $file=<FILE>)
  14. {
  15.     if($file=~/<title>(.+?)<\/title>/)
  16.     {
  17.         print MYTOPIC &stemming($file);
  18.  
  19.     }
  20.     else
  21.     {
  22.        print MYTOPIC $file;
  23.     }                              
  24. }
  25.  
  26.  
  27.  
  28. sub stemming
  29. {
  30.      my $in = shift;
  31.     my $out = '';
  32.  
  33.     foreach my $token (split /[ ,.:;?!]/, $in)
  34.     {
  35.  
  36.           my @stems = $stemmer->stemWord($token);
  37.  
  38.              $out .= $stems[0] . ' ';
  39.  
  40.     }
  41.      return $out;
  42. }
  43. close FILE;
  44. close MYTOPIC;
  45.  
Thanks a lot!
Oct 23 '08 #1
Share this Question
Share on Google+
15 Replies


numberwhun
Expert Mod 2.5K+
P: 3,503
Well, you aren't slurping the file into an array so the size of the file shouldn't make a difference so much.

It makes me wonder how much system memory you have and how many programs are running on your machine. If that it not it, then you will probably need to take this to the PerlBuilder forums as it would be outside of the realms of Perl coding.

Regards,

Jeff

**Note: are you trying to run this from Perl Builder? If so, why not run the script from the command line instead? I personally never trust a program to run my Perl script. I would rather the interpreter do it directly, without a middle man so to speak. As you can see, they always seem to find a way to cause issues.

Regards,

Jeff
Oct 23 '08 #2

KevinADC
Expert 2.5K+
P: 4,059
Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

That script looks like it would use very little memory so I don't know why its running out of memeory unless that module you are using has some known issues. WordNet::QueryData queries a database, that is possibly where the memory problem is happening. Not sure what to suggest though.
Oct 23 '08 #3

numberwhun
Expert Mod 2.5K+
P: 3,503
Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.
Oh, ok. Sorry, wasn't aware. I tend to hand code everything and run from the command line. If I do use an IDE, I use either Eclipse (with EPIC plugin for Perl) or Komodo. Both are awesome.

Well, now I know. :)
Oct 23 '08 #4

Ganon11
Expert 2.5K+
P: 3,652
Oh, ok. Sorry, wasn't aware. I tend to hand code everything and run from the command line. If I do use an IDE, I use either Eclipse (with EPIC plugin for Perl) or Komodo. Both are awesome.

Well, now I know. :)
I'm with you, Numberwhun. Notepad++ to edit, cmd.exe to run.
Oct 23 '08 #5

KevinADC
Expert 2.5K+
P: 4,059
Writing simple scripts in a text editor is fine, but using an IDE is essential to any complex code and even not so complex. You can run syntax checks, pipe data in/out of a script (command line or CGI), write and test code on the fly, use the perl debugger in the IDE, step over code, step into code, run timers , pause a script, see and jump to any subroutine in a single glance (may vary IDE to IDE) see all the variables used in a script (may vary IDE to IDE), and much more.

With the IDE you can open multiple scripts and select the one to run and check the output too, all on the fly if you wanted to.

Make your life easier and use an IDE.

In Perl Builder all the builtin perl functions display a basic syntax usage next to the function if you want it to. So you see something like:

split

(this displays in a little window next to the function)
split /PATTERN/,EXPR,LIMIT
split /PATTERN/,EXPR
split /PATTERN/

No need to look at the documentation if you foget the simple usage syntax. That right there is such a big time saver. And if you wanted to read the entire documentation you just click on the little window and a bigger window pops up with the full perl manpage for the function.
Oct 23 '08 #6

numberwhun
Expert Mod 2.5K+
P: 3,503
Wow, didn't mean to kick you up onto your soap box (j/k). I never said I didn't use an IDE. Komodo has always been my editor of choice but have been playing more with Epic (Eclipse) due to its interesting regex tester.

I agree with you that an IDE is a very useful part of Perl coding, but its not something you cannot live without. The software I support at work is all supported without an IDE, using GVim alone. Some of the files that we make additions to are several thousand lines or more in length.

Regards,

Jeff
Oct 23 '08 #7

Icecrack
Expert 100+
P: 174
i use Crimson Editor for my Perl Coding + other Languages. (FREE WARE)
i did use Perl Builder
Oct 23 '08 #8

KevinADC
Expert 2.5K+
P: 4,059
Did I forget to mention I work for PerlBuilder? J/K..... I just had too much coffee this morning.
Oct 24 '08 #9

Ganon11
Expert 2.5K+
P: 3,652
Did I forget to mention I work for PerlBuilder? J/K..... I just had too much coffee this morning.
*Dare I ask?* What IDE do you suggest? Is PerlBuilder free?
Oct 24 '08 #10

P: 30
Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

That script looks like it would use very little memory so I don't know why its running out of memeory unless that module you are using has some known issues. WordNet::QueryData queries a database, that is possibly where the memory problem is happening. Not sure what to suggest though.
It has no problem when I just convert one word to its stemming words, but when I try to stemming a number of words from a file, the "out of memory" occurs. I think its probably because the wordnet's database 2. I dont know much about how the database works, hope anyone can give me more suggestion.

PS: You guys are friendly and funny, thank you!
Oct 24 '08 #11

P: 30
Since there is no WordNet 3.0 for windows, I download the WordNet 2.1 and change its name to WordNet 3.0... As I said, there was no problem to convert one word to its semming word.
Oct 24 '08 #12

P: 30
I find a problem in my program.

Expand|Select|Wrap|Line Numbers
  1. if($file=~/<title>(.+?)<\/title>/)
  2.  
I thinks this sentence shoud pick the content between the tag <title></title>,
not containing the tag itself. But I test my program and I find the tag is contained in my first split word. For example, <title>perl bulider</title>, the first word is "<title>perl" not "perl"
Oct 24 '08 #13

KevinADC
Expert 2.5K+
P: 4,059
The regexp is probably OK:

Expand|Select|Wrap|Line Numbers
  1. my $file = "test <title>Perl Builder<\/title> test";
  2. if ($file =~ /<title>(.+?)<\/title>/) {
  3.    print $1;
  4. }
I think the problem is here:

Expand|Select|Wrap|Line Numbers
  1. foreach my $token (split /[ ,.:;?!]/, $in)
There is nothing in the above regexp (split) to split the string into tokens except a space. So it gets split into "<title>Perl" and "Builder<\/title>".
Oct 24 '08 #14

P: 30
The regexp is probably OK:

Expand|Select|Wrap|Line Numbers
  1. my $file = "test <title>Perl Builder<\/title> test";
  2. if ($file =~ /<title>(.+?)<\/title>/) {
  3.    print $1;
  4. }
I think the problem is here:

Expand|Select|Wrap|Line Numbers
  1. foreach my $token (split /[ ,.:;?!]/, $in)
There is nothing in the above regexp (split) to split the string into tokens except a space. So it gets split into "<title>Perl" and "Builder<\/title>".
I see your point, you are right! Thank you very much!!
Oct 24 '08 #15

KevinADC
Expert 2.5K+
P: 4,059
I see your point, you are right! Thank you very much!!
You're welcome
Oct 24 '08 #16

Post your reply

Sign in to post your reply or Sign up for a free account.