Connecting Tech Pros Worldwide Help | Site Map
bytes > topic > php > answers

How do I read from one file and write to another ?

  #1  
Old July 5th, 2009, 05:56 AM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165
Hello,

I am not very experienced with file handling, so I am
asking for a little help with this.

I have a file called "geotext_1a.csv"
and it is a csv file looking like this:

"2.6.190.56","2.6.190.63","33996344","33996351","G B","United Kingdom"
"3.0.0.0","4.17.135.31","50331648","68257567","US" ,"United States"
"4.17.135.32","4.17.135.63","68257568","68257599", "CA","Canada"


I want to take some data from it and write a file called geo_01.txt

and will look like this :

33996344,33996351,United Kingdom
50331648,68257567,United States
68257568,68257599,Canada


Now I have written some code but I used
the file function to put the contents into an array.

As the file is a few megabytes big, this has caused memory
failure problems.

This is what I have:
Expand|Select|Wrap|Line Numbers
  1. <?php 
  2. $file_array = file('geotext_1a.csv');
  3. $data = array();
  4. $i = 0;
  5.  
  6. foreach ($file_array as $row)
  7.  {
  8.    $row = str_replace('"','',$row); // get rid of double quotes
  9.      $cells = explode(",",$row);
  10.      $data[$i]['trash1'] = $cells[0];
  11.    $data[$i]['trash2'] = $cells[1];
  12.    $data[$i]['ip1'] = $cells[2];
  13.    $data[$i]['ip2'] = $cells[3];
  14.    $data[$i]['trash3'] = $cells[4];
  15.    $data[$i]['country'] = $cells[5];
  16.      $i++;
  17.   }
  18.  
  19. echo '<pre>';
  20. print_r($data);
  21. echo '</pre>'; 
  22.  
  23. ?>  
So my question is

Does processing the file as a stream use
less memory ?

i.e. $handle = fopen("$file","r")

Because I am not familiar with using this stuff
I don't know how to do this

Any guidance much appreciated

Many thanks
  #2  
Old July 5th, 2009, 07:38 AM
Newbie
  
Join Date: May 2007
Location: Perth, Western Australia
Posts: 10

re: How do I read from one file and write to another ?

Which stage of the code execution is your memory problem occurring at?
  #3  
Old July 5th, 2009, 11:59 AM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

Actually, when I run this , its seems to be working until it gets to a
line and this fatal error occurs:

Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 13 bytes) in /home/ew78gt/public_html/sys/convert.php on line 9
  #4  
Old July 5th, 2009, 05:00 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

Try this:

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. function read_csv_file($file)
  3. {
  4.     $fp = fopen($file, "r");
  5.     $stream = fread($fp, filesize($file));
  6.     $stream = str_replace("\"","",$stream);
  7.     $rows = explode("\n", $stream);
  8.     $data = array();
  9.     for($i = 0; $i<=count($rows)-1; $i++)
  10.     {
  11.         $cells = explode(",",$rows[$i]);
  12.         $data[$i] = array("trash1" => $cells[0], "trash2" => $cells[1], "ip1" => $cells[2], "ip2" => $cells[3], "trash3" => $cells[4], "country" => $cells[5]);
  13.     }
  14.     return $data;
  15. }
  16.  
  17. echo '<pre>';
  18. print_r(read_csv_file("test.csv"));
  19. echo '</pre>'; 
  20. ?>  
hope this helps you..
  #5  
Old July 6th, 2009, 07:05 AM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

Thanks for your suggestion.
I ran it on my server and I got this output:


Notice: Undefined offset: 1 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 2 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 3 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 4 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 5 in /home/ew78gt/public_html/sys/con_01.php on line 12



Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 15 bytes) in /home/ew78gt/public_html/sys/con_01.php on line 11
  #6  
Old July 6th, 2009, 07:28 AM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

Instead of putting the whole file into memory,
can I not work on the file just one line at a time ?

ı.e.

1) open the file to read,
2) open file to write,

3) read first row
4) take out double quotes
5) explode row into 6 variables
6) write desired 3 variables to row in new file

7) loop and read next row until end of file.
8) close files


So:

$file1 = "geotext_1a.csv";
$file2 = "geo_new".txt";

$fp1 = fopen($file1, "r");
$fp2 = fopen($file2, "w");

while not-end-of-file1
3) read first row
$row = str_replace('"','',$row);
$cells = explode(",",$row);
write to file2 on first row $cells[2],$cells[3], $cells[5]
}

close files

If someone can help me fill in the bits
that I am missing shouldn't this thenl work ok without
memory problems ?
  #7  
Old July 6th, 2009, 11:19 AM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

it's possible, just upload the large file (CSV) and give me the link of uploaded file, i will make the code for you
  #8  
Old July 6th, 2009, 01:27 PM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

The file is here:

http://www.expert-world.net/sys/geotext_1a.csv

feel free to download it.
  #9  
Old July 6th, 2009, 03:55 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

Ok, i downloaded your file and test it with the previous script that i wrote for you and it worked correctly, it seems that you have something wrong with writing columns in your new file,
i changed the script and added another function that works great with your CSV file (i have already test that)

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. function read_csv_file($file, $newfile)
  3. {
  4.     $fp = fopen($file, "r");
  5.     $stream = fread($fp, filesize($file));
  6.     $stream = str_replace("\"","",$stream);
  7.     $rows = explode("\n", $stream);
  8.     $data = array();
  9.     for($i = 0; $i<=count($rows)-1; $i++)
  10.     {
  11.         if(strpos($rows[$i], ","))
  12.         {
  13.             $cells = explode(",",$rows[$i]);
  14.             if(count($cells>=6))
  15.                 write_to_file($newfile, $cells[0], $cells[1], $cells[2]); // You can replace your own columns here (To write to the new file)
  16.         }
  17.     }
  18. }
  19.  
  20. function write_to_file($file, $col1, $col2, $col3)
  21. {
  22.     $stream = $col1.",".$col2.",".$col3;
  23.     if(strlen($stream)>5)
  24.     {
  25.         $fileExists = file_exists($file);
  26.         $fp = fopen($file, "a");
  27.         if($fileExists) $stream = "\n" . $stream;
  28.         fwrite($fp, $stream, strlen($stream));
  29.         fclose($fp);
  30.     }
  31. }
  32.  
  33. // export columns from the file
  34. read_csv_file("sourceFile.csv", "newFile.txt");
  35. ?> 
  36.  
remember 1 thing:
when i checked your CSV file, there was some empty row, that's why you face to those warnings (Notice: Undefined offset: 1 in /home/ew78gt ..)
you don't need to remove the empty lines, the script will ignore them.

if you had any questions, feel free to ask me ;)
  #10  
Old July 6th, 2009, 08:30 PM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

Thanks very much,

I really appreciate the time.

While you doing that I have been reading the manual and
I found these two commands:
fgetcsv() and fputcsv()

So I have written this:

But do not know if it will work.
May be it won't ignore those problem lines ?

What do you think ?


Expand|Select|Wrap|Line Numbers
  1. <?php
  2. $row = 1;
  3. $handle1 = fopen("geotext_1a.csv", "r");
  4. $handle2 = fopen("geo_sm_1a.csv", "w");
  5. while (($data = fgetcsv($handle1,  ",")) !== FALSE) {
  6.     $row = str_replace('"','',$data );
  7.     $cells = explode(",",$row);
  8.         $new_arr = array('$cells[2]', '$cells[3]', '$cells[5]'); 
  9.  
  10.     fputcsv($handle2,$new_arr);
  11.    }
  12. fclose($handle1);
  13. fclose($handle2);
  14. ?>
  15.  
  #11  
Old July 6th, 2009, 09:58 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

alright, i tried to test your script, but it doesn't work for me
if it works for you, i just added few lines to make sure your script will ignore the empty lines:

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. $row = 1;
  3. $handle1 = fopen("geotext_1a.csv", "r");
  4. $handle2 = fopen("geo_sm_1a.csv", "w");
  5. while (($data = fgetcsv($handle1,  ",")) !== FALSE)
  6. {
  7.     if(strlen($data)>5)
  8.     {
  9.         $row = str_replace('"','',$data);
  10.         $cells = explode(",",$row);
  11.         if(count($cells) >= 6)
  12.         {
  13.             $new_arr = array($cells[2], $cells[3], $cells[5]);     
  14.             fputcsv($handle2,$new_arr);
  15.         }
  16.     }
  17. }
  18. fclose($handle1);
  19. fclose($handle2);
  20. ?> 
  21.  
remember this:
as far as i know you can use variables in double quotations (") like "$str" but about the single quotation ('), it will ignore the variable, it will write exact string like "$str" not the value of that..

so i changed [array('$cells[2]', '$cells[3]', '$cells[5]')] to [array($cells[2], $cells[3], $cells[5])]

fgetcsv() is exactly like fgets(), but it will explode them by given character
i'm still recommend you to use the previous script that i wrote for you, because this script didn't work for me, i dont know about you,

if previous script didn't work for you, tell me to re write the script with this functions ;)
  #12  
Old July 7th, 2009, 09:29 AM
Markus's Avatar
Moderator
  
Join Date: Jun 2007
Location: York, England, with wolves.
Posts: 4,862
Provided Answers: 9

re: How do I read from one file and write to another ?

You can read a file, line by line, by using the file() function - see php.net/file
  #13  
Old July 7th, 2009, 09:49 AM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

Quote:
Originally Posted by Markus View Post
You can read a file, line by line, by using the file() function - see php.net/file
We know about that, he have problem with writing some columns of the main file in a new file
  #14  
Old July 7th, 2009, 01:45 PM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

Hi pezhvak,

Thanks again,

I was hoping that you could get the script that I wrote to
work. - You seem to know more about this stuff than me.

The reason I liked my script is because it is less code.

What do you you think I did wrong?

This is my first time trying to change a file like this !!

I will use your script, (thanks)
but I think you how it is... I would just
like to know what is wrong with the scrpt I wrote !!

Thanks for your help.
  #15  
Old July 7th, 2009, 02:31 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

as your request i checked your script again,

Your problem was:

1- in the [while] loop:
all of the conditions will be converted to true or false, so we don't need !==false in our conditions (this will not effect your script, i said that just to know ;) )

2- actually in this part ($data = fgetcsv($handle1, ",")), $data will convert to an array and you cannot replace double quotations like you wrote:

Expand|Select|Wrap|Line Numbers
  1. $row = str_replace('"','',$data);  // it's completely wrong
3- as i said, fgetcsv will automatically split your string by given character.
so when you use $data = fgetcsv($handle1, "," ) in the [while] loop, it means that while until EOF (eof in the given file handle)

fgetcsv will read your CSV file line by line and it will split the line by given character (argument 2) and will return an array

so:
Expand|Select|Wrap|Line Numbers
  1. $cells = explode(",",$row); // this is completely wrong too!
4- why you wrote this line ($row = 1;)?? ;)


fixed code:
Expand|Select|Wrap|Line Numbers
  1. <?php
  2. $handle1 = fopen("geotext_1a.csv", "r");
  3. $handle2 = fopen("geo_sm_1a.csv", "w");
  4. while ($data = fgetcsv($handle1,  ","))
  5. {
  6.     $row = $data[2]."[,]".$data[3]."[,]".trim($data[5]);
  7.     if(strlen($row)>10)
  8.     {
  9.         $row = str_replace('"','',$row);
  10.         $row = explode("[,]", $row);
  11.         fputcsv($handle2, $row);
  12.     }
  13. }
  14. fclose($handle1);
  15. fclose($handle2);
  16. ?> 
  17.  
remember:
[fputcsv] will add double quotations automatically for those indexes of given array that contains [space] character, so if you dont want it, you must use another function to write the line to your file

so... it was all about your script,

i agree with you, this script is much simpler that the one i gave to you, it will work great, exactly like the script that i sent before...

UPDATE:
i added few lines to the script, because i saw some of the countries like my country have [,] in their name, script will split the string by [,] and return an array with 4 indexes, and it's wrong

ex: 209868800,209869055,Iran, Islamic Republic of
NOTE: you don't really need to save the name like this!! (ignore the "Islamic Republic of") "Iran" is enough!!:D

alright another change that i made is trim(),
i add this function because i saw some columns (country column) in the final file that contains [space] before their name... so i add trim() function to remove those :)

NOTE: trim() will remove space (space is default character) from begin and end of the string;
  #16  
Old July 8th, 2009, 11:50 AM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

Thanks VERY much.

I appreciate your explanation,
and using my script as a basis.

A combined effort to end up with a nice short
script :)

I ran it and it created the new file. :) :)

The output does give quite a few notices.

Is this because of those commas in the country description ?
Iran, Congo and Tanzania all have them.

Is there an easy way to truncate those country names at the comma ?

Here are the notices I get:
( these repeat many times)

Notice: Undefined offset: 2 in /home/ew78gt/public_html/sys/con_01.php on line 6

Notice: Undefined offset: 3 in /home/ew78gt/public_html/sys/con_01.php on line 6

Notice: Undefined offset: 5 in /home/ew78gt/public_html/sys/con_01.php on line 6

Notice: Undefined offset: 2 in /home/ew78gt/public_html/sys/con_01.php on line 6

Notice: Undefined offset: 3 in /home/ew78gt/public_html/sys/con_01.php on line 6

Notice: Undefined offset: 5 in /home/ew78gt/public_html/sys/con_01.php on line 6

Can we get rid of these notices as well as the extended country name parts ?

Thanks again.
  #17  
Old July 8th, 2009, 12:54 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

Aha! i think i know what is the problem!!
Do you remember the empty lines in your CSV file?
that's the problem, as i said, fgetcsv will explode the line by ',' character.
in other hand there is no ',' character in the empty lines, so we don't have $data[1-5] indexes, so it will show those notices,
So, i update the script again:

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. $handle1 = fopen("geotext_1a.csv", "r");
  3. $handle2 = fopen("geo_sm_1a.csv", "w");
  4. while ($data = fgetcsv($handle1,  ","))
  5. {
  6.     if(count($data) >= 5)
  7.     {
  8.         $row = $data[2]."[,]".$data[3]."[,]".trim($data[5]);
  9.         if(strlen($row)>10)
  10.         {
  11.             $row = str_replace('"','',$row);
  12.             $row = explode("[,]", $row);
  13.             fputcsv($handle2, $row);
  14.         }
  15.     }
  16. }
  17. fclose($handle1);
  18. fclose($handle2);
  19. ?> 
  20.  
i add this part to the script:
Expand|Select|Wrap|Line Numbers
  1. if(count($data) >= 5) //so, we will check if there are more than 4 indexes, if yes, we will write it to second CSV file
  2.  
hope this resolve your problem ;)
  #18  
Old July 8th, 2009, 03:39 PM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

You are one smart cookie :)

ran it again and no notices.

so to get trim off these extra bits in the
country name:

would I insert this:

$data[5] =strstr($data[5], ',', true);


so:

<?php
$handle1 = fopen("geotext_1a.csv", "r");
$handle2 = fopen("geo_sm_1a.csv", "w");
while ($data = fgetcsv($handle1, ",")) {
if(count($data) >=5) {
$data[5] =strstr($data[5], ',', true);
$row = $data[2]."[,]".$data[3]."[,]".trim($data[5]);
if(strlen($row)>10) {
$row = str_replace('"','',$row);
$row = explode("[,]", $row);
fputcsv($handle2, $row);
}
}
}
fclose($handle1);
fclose($handle2);
?>

Well I tried it and I get :

Warning: Wrong parameter count for strstr() in /home/ew78gt/public_html/sys/con_01.php on line 6

This might be because I have php 5.2 on my server
and not 5.3 as required.

How else can I cut everything after a comma in
$data[5] - be good to get rid of the comma as well :)
  #19  
Old July 8th, 2009, 04:46 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

Alright, here you are:

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. $handle1 = fopen("geotext_1a.csv", "r");
  3. $handle2 = fopen("geo_sm_1a.csv", "w");
  4. while ($data = fgetcsv($handle1,  ","))
  5. {
  6.     if(count($data) >= 5)
  7.     {
  8.         $country = strpos($data[5], ",") ? substr($data[5],0,strpos($data[5], ",")) : $data[5];
  9.         $row = $data[2]."[,]".$data[3]."[,]".trim($country);
  10.         if(strlen($row)>10)
  11.         {
  12.             $row = str_replace('"','',$row);
  13.             $row = explode("[,]", $row);
  14.             fputcsv($handle2, $row);
  15.         }
  16.     }
  17. }
  18. fclose($handle1);
  19. fclose($handle2);
  20. ?> 
  21.  
this line will remove everything after comma in country column:
Expand|Select|Wrap|Line Numbers
  1. $country = strpos($data[5], ",") ? substr($data[5],0,strpos($data[5], ",")) : $data[5];
  2.  
  #20  
Old July 9th, 2009, 05:38 AM
Familiar Sight
  
Join Date: Jan 2009
Posts: 165

re: How do I read from one file and write to another ?

I tried what you suggested, but it did not work.

Then, I rewrote it in a style I understand:

Expand|Select|Wrap|Line Numbers
  1. if (strpos($data[5], ",")) { 
  2.  $country = substr($data[5],0,strpos($data[5], ","));
  3.  }
  4. else {
  5.   $country = $data[5];
  6. }
  7. $row = $data[2]."[,]".$data[3]."[,]".trim($country);  
I think it is just the long-nad version of what you wrote.

It looks like it should work to me - but it does not.


SORRY

MY Mistake - it works fine now :)

Thanks again
  #21  
Old July 9th, 2009, 10:21 AM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

Your welcome ;)

if you had any questions, feel free to ask me :)
  #22  
Old August 20th, 2009, 02:03 PM
Newbie
  
Join Date: Aug 2009
Posts: 2

re: How do I read from one file and write to another ?

Hello,

I have following code to read csv file, do some manipulations(i.e. add new records), and write it to new csv file. When i execute it on small files it works perfectly but when i use it for bigger files like 9MB following error occurs:

Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 35 bytes) in D:\projects\wamp\www\data_handling\data_handling.p hp on line 16.

please help me out solving thios problem

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. set_time_limit(600);
  3.  
  4. if($_FILES)
  5. {
  6.     $filename    =    $_FILES['file']['name'];
  7.  
  8.     move_uploaded_file($_FILES['file']['tmp_name'], $_FILES['file']['name']);
  9.  
  10.     $handle = fopen("$filename", 'r');
  11.     $records    =    array();
  12.     $i = 0;
  13.     $csv_output    =    '';
  14.     while(($data = fgetcsv($handle, 4096, ",")) !== false)
  15.     {
  16.         $records[$i]    =    $data;
  17.         $i++;
  18.     }
  19.  
  20.     fclose($handle);
  21.  
  22.     for($j=0; $j<$i; $j++)
  23.     {
  24.         for($k=0; $k<=26; $k++)
  25.         {
  26.             if($k == 7)
  27.                 continue;
  28.             else if($j==0)    
  29.                 $csv_output .= $records[$j][$k].",";
  30.  
  31.             if($j>0)
  32.             {
  33.                 $start    =     $records[$j][6];
  34.                 $end    =    $records[$j][7];
  35.  
  36.                 for($l = $start; $l<$end; $l++)
  37.                 {
  38.                     for($k=0; $k<=26; $k++)
  39.                     {
  40.                         if($k == 7)
  41.                             continue;
  42.                         if($j>0 && $k == 6)
  43.                         {
  44.                             $csv_output .= $l.",";
  45.                         }
  46.                         else
  47.                         {
  48.                             $csv_output .= $records[$j][$k].",";
  49.                         }
  50.                     }
  51.  
  52.                     $csv_output .= "\n";
  53.                 }
  54.             }
  55.         }
  56.         if($j == 0)
  57.         $csv_output .= "\n";
  58.     }
  59.     //print $csv_output;
  60.     $export_file = "output_".$filename;
  61.     $fp = fopen($export_file, "wb");
  62.     if (!is_resource($fp))
  63.     {
  64.         die("Cannot open $export_file");
  65.     }
  66.  
  67.     fwrite($fp, $csv_output);
  68.     fclose($fp);
  69.  
  70.     exit;
  71. }
  72. ?>
Last edited by Dormilich; August 20th, 2009 at 02:30 PM. Reason: Please use [code] tags when posting code
  #23  
Old August 20th, 2009, 02:53 PM
Banned
  
Join Date: Jul 2009
Posts: 17

re: How do I read from one file and write to another ?

it's all about your PHP.ini file, you have to change your memory_limit & post_max_size and also upload_max_filesize that is required to be changed, in default the value of upload_max_filesize is 5mb, change it to 2000mb or more... so your script will work again
  #24  
Old August 21st, 2009, 01:11 PM
Newbie
  
Join Date: Aug 2009
Posts: 2

re: How do I read from one file and write to another ?

Thanks a lot. Its done by increasing memory size to 300M.
Thank you so much.
Reply

« previous question | next question »

/bytes/development

/bytes/it

/bytes/about

We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights. Get the best answers to your questions from over 225,702 network members.

dev questions:
algorithms | asp | asp.net | c/c++ | coldfusion | c# | flash | html/css | java | javascript |
mobile dev | .net | perl | php | python | ruby | software dev | vb.net | visual basic | xml
it questions:
access | apache | careers | db2 | iis | macosx | mysql | networking | oracle | postgresql | sql server | unix | windows

About Bytes | Copyright 1995-2009 BYTES. All rights Reserved