473,566 Members | 3,040 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How do I read from one file and write to another ?

290 Contributor
Hello,

I am not very experienced with file handling, so I am
asking for a little help with this.

I have a file called "geotext_1a.csv "
and it is a csv file looking like this:

"2.6.190.56","2 .6.190.63","339 96344","3399635 1","GB","Uni ted Kingdom"
"3.0.0.0","4.17 .135.31","50331 648","68257567" ,"US","Unite d States"
"4.17.135.32"," 4.17.135.63","6 8257568","68257 599","CA","Cana da"


I want to take some data from it and write a file called geo_01.txt

and will look like this :

33996344,339963 51,United Kingdom
50331648,682575 67,United States
68257568,682575 99,Canada


Now I have written some code but I used
the file function to put the contents into an array.

As the file is a few megabytes big, this has caused memory
failure problems.

This is what I have:
Expand|Select|Wrap|Line Numbers
  1. <?php 
  2. $file_array = file('geotext_1a.csv');
  3. $data = array();
  4. $i = 0;
  5.  
  6. foreach ($file_array as $row)
  7.  {
  8.    $row = str_replace('"','',$row); // get rid of double quotes
  9.      $cells = explode(",",$row);
  10.      $data[$i]['trash1'] = $cells[0];
  11.    $data[$i]['trash2'] = $cells[1];
  12.    $data[$i]['ip1'] = $cells[2];
  13.    $data[$i]['ip2'] = $cells[3];
  14.    $data[$i]['trash3'] = $cells[4];
  15.    $data[$i]['country'] = $cells[5];
  16.      $i++;
  17.   }
  18.  
  19. echo '<pre>';
  20. print_r($data);
  21. echo '</pre>'; 
  22.  
  23. ?>  
So my question is

Does processing the file as a stream use
less memory ?

i.e. $handle = fopen("$file"," r")

Because I am not familiar with using this stuff
I don't know how to do this

Any guidance much appreciated

Many thanks
Jul 5 '09 #1
23 7266
Cmaza
16 New Member
Which stage of the code execution is your memory problem occurring at?
Jul 5 '09 #2
jeddiki
290 Contributor
Actually, when I run this , its seems to be working until it gets to a
line and this fatal error occurs:

Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 13 bytes) in /home/ew78gt/public_html/sys/convert.php on line 9
Jul 5 '09 #3
pezhvak
17 New Member
Try this:

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. function read_csv_file($file)
  3. {
  4.     $fp = fopen($file, "r");
  5.     $stream = fread($fp, filesize($file));
  6.     $stream = str_replace("\"","",$stream);
  7.     $rows = explode("\n", $stream);
  8.     $data = array();
  9.     for($i = 0; $i<=count($rows)-1; $i++)
  10.     {
  11.         $cells = explode(",",$rows[$i]);
  12.         $data[$i] = array("trash1" => $cells[0], "trash2" => $cells[1], "ip1" => $cells[2], "ip2" => $cells[3], "trash3" => $cells[4], "country" => $cells[5]);
  13.     }
  14.     return $data;
  15. }
  16.  
  17. echo '<pre>';
  18. print_r(read_csv_file("test.csv"));
  19. echo '</pre>'; 
  20. ?>  
hope this helps you..
Jul 5 '09 #4
jeddiki
290 Contributor
Thanks for your suggestion.
I ran it on my server and I got this output:


Notice: Undefined offset: 1 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 2 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 3 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 4 in /home/ew78gt/public_html/sys/con_01.php on line 12



Notice: Undefined offset: 5 in /home/ew78gt/public_html/sys/con_01.php on line 12



Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 15 bytes) in /home/ew78gt/public_html/sys/con_01.php on line 11
Jul 6 '09 #5
jeddiki
290 Contributor
Instead of putting the whole file into memory,
can I not work on the file just one line at a time ?

ı.e.

1) open the file to read,
2) open file to write,

3) read first row
4) take out double quotes
5) explode row into 6 variables
6) write desired 3 variables to row in new file

7) loop and read next row until end of file.
8) close files


So:

$file1 = "geotext_1a.csv ";
$file2 = "geo_new".t xt";

$fp1 = fopen($file1, "r");
$fp2 = fopen($file2, "w");

while not-end-of-file1
3) read first row
$row = str_replace('"' ,'',$row);
$cells = explode(",",$ro w);
write to file2 on first row $cells[2],$cells[3], $cells[5]
}

close files

If someone can help me fill in the bits
that I am missing shouldn't this thenl work ok without
memory problems ?
Jul 6 '09 #6
pezhvak
17 New Member
it's possible, just upload the large file (CSV) and give me the link of uploaded file, i will make the code for you
Jul 6 '09 #7
jeddiki
290 Contributor
The file is here:

http://www.expert-world.net/sys/geotext_1a.csv

feel free to download it.
Jul 6 '09 #8
pezhvak
17 New Member
Ok, i downloaded your file and test it with the previous script that i wrote for you and it worked correctly, it seems that you have something wrong with writing columns in your new file,
i changed the script and added another function that works great with your CSV file (i have already test that)

Expand|Select|Wrap|Line Numbers
  1. <?php
  2. function read_csv_file($file, $newfile)
  3. {
  4.     $fp = fopen($file, "r");
  5.     $stream = fread($fp, filesize($file));
  6.     $stream = str_replace("\"","",$stream);
  7.     $rows = explode("\n", $stream);
  8.     $data = array();
  9.     for($i = 0; $i<=count($rows)-1; $i++)
  10.     {
  11.         if(strpos($rows[$i], ","))
  12.         {
  13.             $cells = explode(",",$rows[$i]);
  14.             if(count($cells>=6))
  15.                 write_to_file($newfile, $cells[0], $cells[1], $cells[2]); // You can replace your own columns here (To write to the new file)
  16.         }
  17.     }
  18. }
  19.  
  20. function write_to_file($file, $col1, $col2, $col3)
  21. {
  22.     $stream = $col1.",".$col2.",".$col3;
  23.     if(strlen($stream)>5)
  24.     {
  25.         $fileExists = file_exists($file);
  26.         $fp = fopen($file, "a");
  27.         if($fileExists) $stream = "\n" . $stream;
  28.         fwrite($fp, $stream, strlen($stream));
  29.         fclose($fp);
  30.     }
  31. }
  32.  
  33. // export columns from the file
  34. read_csv_file("sourceFile.csv", "newFile.txt");
  35. ?> 
  36.  
remember 1 thing:
when i checked your CSV file, there was some empty row, that's why you face to those warnings (Notice: Undefined offset: 1 in /home/ew78gt ..)
you don't need to remove the empty lines, the script will ignore them.

if you had any questions, feel free to ask me ;)
Jul 6 '09 #9
jeddiki
290 Contributor
Thanks very much,

I really appreciate the time.

While you doing that I have been reading the manual and
I found these two commands:
fgetcsv() and fputcsv()

So I have written this:

But do not know if it will work.
May be it won't ignore those problem lines ?

What do you think ?


Expand|Select|Wrap|Line Numbers
  1. <?php
  2. $row = 1;
  3. $handle1 = fopen("geotext_1a.csv", "r");
  4. $handle2 = fopen("geo_sm_1a.csv", "w");
  5. while (($data = fgetcsv($handle1,  ",")) !== FALSE) {
  6.     $row = str_replace('"','',$data );
  7.     $cells = explode(",",$row);
  8.         $new_arr = array('$cells[2]', '$cells[3]', '$cells[5]'); 
  9.  
  10.     fputcsv($handle2,$new_arr);
  11.    }
  12. fclose($handle1);
  13. fclose($handle2);
  14. ?>
  15.  
Jul 6 '09 #10

Sign in to post your reply or Sign up for a free account.

Similar topics

22
13218
by: Jason Heyes | last post by:
Does this function need to call eof after the while-loop to be correct? bool read_file(std::string name, std::string &s) { std::ifstream in(name.c_str()); if (!in.is_open()) return false; char c; std::string str;
1
6729
by: cnu | last post by:
My program generates a log file for every event that happens in the program. So, I open the file and keep it open till the end. This is how I open the file for writing: <CODE> public CLogHandler() { this.m_fsLog = new FileStream(strTodaysLogFile, System.IO.FileMode.Append, System.IO.FileAccess.Write, System.IO.FileShare.Read);...
5
3154
by: Martin Svensson | last post by:
Hello! I need some help/recommendations on how to do the following. I have a program that writes an IP address two control numbers and a date to file, on one line. It's a basic text file and it looks like this: ----- beginning of file ----- 192.168.0.1 0 0 2004-08-01 192.168.0.2 1 0 2004-08-21 192.168.0.4 0 1 2004-08-12
3
2741
by: Bill Cohagan | last post by:
I'm writing a console app in c# and am encountering a strange problem. I'm trying to use redirection of the standard input stream to read input from a (xml) file. The following code snippet is from this app: =============================== static void Main(string args) { if (args.Length > 0) Console.SetIn(new StreamReader(args)); //executes...
2
4058
by: GB | last post by:
Hi Everybody! I have 2 different processes/application. One is writing to a file and another is reading from it. For some reason the code doesnt seems to work and gives mscorlib.dll IOException error "This file is being used by another process". Both the applications are in C#. P.S. Even if I try to open the file with NotePad (while my...
10
2526
by: Tibby | last post by:
I need to read/write not only text files, but binary as well. It seems like on binary files, it doesn't right the last 10% of the file. -- Thanks --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.538 / Virus Database: 333 - Release Date: 11/10/2003
5
5088
by: Sumana | last post by:
Hi All, We developed our project on VC++.Net console application to create image of disk and to write the image We are having problem with reading and writing the sector beyond 6GB Disk or Partition we are using ReadFile , WriteFile and setFilePointerEx to read and write the sectors and we are reading/writing 102400 sectors together, even...
8
2326
by: dosworldguy | last post by:
I have been having a very peculiar issue from a long time. I have an application where multiple clients read from a shared set of files. When a record is changed, sometimes the win9x clients still read the old data (if it was read earlier) and this is causing data corruption. WinNT clients inlcuding windows2000 & XP do not have this issue....
3
8273
by: eholz1 | last post by:
Hello PHP Group, I am having trouble setting permissions correctly so that the magickwand api (php 5.2) can read and write images. I usually read a file from one directory, create a magickwand resource from that file, and transform the image, and save the new image with a new name to a different directory. I have seen that my file and...
4
2290
by: someone28485 | last post by:
hi,i need 2 write code in C#.Net to read a text file with text as 22.13 21.65 25.16 31.22 and so on, i need 2 read this text and then take mean of the 4 values and write that mean result in another file, so far i've only been able to read the whole text from one file and write it to another file,can anyone please help me solve my...
0
7683
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7586
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7899
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
6270
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5215
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3647
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3628
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1204
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
927
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.