469,366 Members | 2,408 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,366 developers. It's quick & easy.

Reading in very large line of text

Hi, I'm fairly new to perl and I'm running into a problem. I have a program that reads in a file line by line, performs a change on the text, and then outputs it to another file. I use "while(my $line =<DATA>) {" to read it in.
Everything works fine but now I have a file which is 1.9 GB, It has only 2 lines in it, and the program is unable to read in a entire line at a time.
I would like to know if there is a different way to read in data that will not blow out my memory.
Thanks
Nov 14 '07 #1
8 3641
numberwhun
3,503 Expert Mod 2GB
Hi, I'm fairly new to perl and I'm running into a problem. I have a program that reads in a file line by line, performs a change on the text, and then outputs it to another file. I use "while(my $line =<DATA>) {" to read it in.
Everything works fine but now I have a file which is 1.9 GB, It has only 2 lines in it, and the program is unable to read in a entire line at a time.
I would like to know if there is a different way to read in data that will not blow out my memory.
Thanks
Well, that depends. Perl is limited by the system memory. Is there any way that the lines could be broken up on to separate lines? Just curious.

Regards,

Jeff
Nov 14 '07 #2
Well, that depends. Perl is limited by the system memory. Is there any way that the lines could be broken up on to separate lines? Just curious.

Regards,

Jeff
No, I have no control over the format of the file I recieve.
Nov 14 '07 #3
jonathan184
154 100+
Just a suggestion maybe find something common in the lines that show up in every file and use that to break the line into multiple lines something like a delimiter and configure the script to add maybe a \n or something to create a new line but you will use it to create a new line. Then do your edit with the script and search for the same delimiter then have the script remove the \n and it should go back to what format you had it as before or just write to another file as the same format.


Like i said just a suggestion
Nov 14 '07 #4
Just a suggestion maybe find something common in the lines that show up in every file and use that to break the line into multiple lines something like a delimiter and configure the script to add maybe a \n or something to create a new line but you will use it to create a new line. Then do your edit with the script and search for the same delimiter then have the script remove the \n and it should go back to what format you had it as before or just write to another file as the same format.


Like i said just a suggestion
I'd like to do this but I don't know how without first reading the file in. Can you give an example of this without first reading the whole line into memory?
Nov 14 '07 #5
KevinADC
4,059 Expert 2GB
set the file to binmode and use the read() function to read in a certain amount of data from the file at a time. Something like:

Expand|Select|Wrap|Line Numbers
  1. my $data;
  2. my $buffer = 1024 * 1024 * 2;
  3. open(IN, 'yourfile') or die "$!";
  4. binmode IN;
  5. while( read( IN, $data, $buffer ) ) {
  6.     do something with $data
  7. }
  8. close IN;
adjust $buffer as needed. In the above code it is 2 megabytes.
Nov 14 '07 #6
Have you tried tie::file module.
This modules allows you to read a file, size doesn't matter without any memory limitations.

Hope you digg into it.

---
Nexis
Nov 14 '07 #7
Have you tried tie::file module.
This modules allows you to read a file, size doesn't matter without any memory limitations.

Hope you digg into it.

---
Nexis
http://startperl.blogspot.com/
thanks, I'll try this, it seems promising.
Nov 14 '07 #8
KevinADC
4,059 Expert 2GB
Let us know what happens with Tie::File. I don't think you will be able to use it for this odd file you have. Tie::File still has to read data into memory at some point in the game so I think you will hit the same wall you did before. You may need to read the file in chunks, as I showed you above. The trick will be getting the output to the new file correct.

Tie::File is good when you want to access the lines of a large file in an array like manner, but I don't think that is what you are doing.
Nov 14 '07 #9

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

15 posts views Thread by djj858 | last post: by
4 posts views Thread by Matthew Crema | last post: by
9 posts views Thread by haibhoang | last post: by
3 posts views Thread by vbnewbie | last post: by
3 posts views Thread by xyz | last post: by
1 post views Thread by CARIGAR | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.