473,799 Members | 3,033 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Parsing large files by line

Does anyone have a suggestion for parsing large files line by line without
loading the entire file into memory first? I don't want to use file()
because the files I'm working with may be multi-gigabyte so loading them
into arrays would be pretty memory intensive.

In Perl, I can do: "while($line=<H ANDLE>){do something with $line}".

Is there an equivalent function in PHP?

Thanks,
Kevin
Aug 15 '05 #1
3 1801
Kevin wrote:
Does anyone have a suggestion for parsing large files line by line without
loading the entire file into memory first? I don't want to use file()
because the files I'm working with may be multi-gigabyte so loading them
into arrays would be pretty memory intensive.

In Perl, I can do: "while($line=<H ANDLE>){do something with $line}".

Is there an equivalent function in PHP?

Thanks,
Kevin


$handle=fopen(. ..);
while($line=fge ts($handle)) {
...
}
fclose($handle) ;

C.
Aug 15 '05 #2
Just what I was looking for, thanks!

- Kevin

"Colin McKinnon" <co************ **@andthis.mms3 .com> wrote in message
news:dd******** ***********@new s.demon.co.uk.. .
Kevin wrote:
Does anyone have a suggestion for parsing large files line by line
without
loading the entire file into memory first? I don't want to use file()
because the files I'm working with may be multi-gigabyte so loading them
into arrays would be pretty memory intensive.

In Perl, I can do: "while($line=<H ANDLE>){do something with $line}".

Is there an equivalent function in PHP?

Thanks,
Kevin


$handle=fopen(. ..);
while($line=fge ts($handle)) {
...
}
fclose($handle) ;

C.

Aug 15 '05 #3
Kevin schrieb:
Does anyone have a suggestion for parsing large files line by line without
loading the entire file into memory first? I don't want to use file()
because the files I'm working with may be multi-gigabyte so loading them
into arrays would be pretty memory intensive.

In Perl, I can do: "while($line=<H ANDLE>){do something with $line}".

Is there an equivalent function in PHP?

Thanks,
Kevin


$handle = fopen("/tmp/inputfile.txt", "r");
while (!feof($handle) ) {
$buffer = fgets($handle, 4096);
echo $buffer;
}
fclose($handle) ;

more at www.php.net/fgets

Markus
Aug 15 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
19470
by: Ratnakar Pedagani | last post by:
Hi, I'm trying to parse the text file, which is of size more than 2mb. I'm using the following sample code Open "c:\sim1.txt" For Input As #1 Do While Not EOF(1) Input #1, Data If (InStr(Data, "Summary")) Then
2
3960
by: Cigdem | last post by:
Hello, I am trying to parse the XML files that the user selects(XML files are on anoher OS400 system called "wkdis3"). But i am permenantly getting that error: Directory0: \\wkdis3\ROOT\home Canonicalpath-Directory4: \\wkdis3\ROOT\home\bwe\ You selected the file named AAA.XML getXmlAlgorithmDocument(): IOException Not logged in
9
4055
by: PedroX | last post by:
Hello: I need to parse some large XML files, and save the data in an Access DB. I was using MSXML 2 and ASP, but it turns out to be extremely slow when then XML documents are like 10 mb in size. It's taking over an hour to parse such sizes!? I don't really need to use ASP or a web server at all because I am parsing all in my own computer. Is there any executable that can do this parsing faster than the way I was doing it?
6
3160
by: ArunPrakash | last post by:
Hi, I have a web application that looks for a particular string in a set of huge files( the files grow into MBs - max i have seen is 30 MB ). ( search using reg expressions ). the string can occur multiple times in a file. whenver the string is found in a line, the whole line must be printed in the output. What i am doing is, 1. traversing each file in the directory 2. In each file, read line by line 3. Match the regular expression. If...
8
2944
by: Eric Anderson | last post by:
I have some files that sit on a FTP server. These files contain data stored in a tab-separated format. I need to download these files and insert/update them in a MySQL database. My current basic strategy is to do the following: 1) Login to the ftp server using the FTP library in PHP 2) Create a variable that acts like a file handle using Stream_Var in PEAR. 3) Use ftp_fget() to read a remote file into this variable (this is so I don't...
7
2515
by: Alan | last post by:
Hi. I have programmed in C++ before, but I`m a couple of years out of practice. I am seeking some advice on getting started on a quickie project. . . . I have to read a 54MB text file and do a pairwise comparison among 2500 items or so in the file. Each of those items have to be compared to every other item. Many of the comparison will only require comparing one field of the items. I will probably sort on this field before I do the...
3
4387
by: toton | last post by:
Hi, I have some ascii files, which are having some formatted text. I want to read some section only from the total file. For that what I am doing is indexing the sections (denoted by .START in the file) with the location. And for a particular section I parse only that section. The file is something like, .... DATAS
1
1768
by: Robert Neville | last post by:
Basically, I want to create a table in html, xml, or xslt; with any number of regular expressions; a script (Perl or Python) which reads each table row (regex and replacement); and performs the replacement on any file name, folder, or text file (e.g. css, php, html). For example, I often rename my mp3 (files); the folder holding the mp3 files; and replace these renamed values in a playlist/m3u/xml file. The table should hold clean...
10
2001
by: len | last post by:
I have created the following program to read a text file which happens to be a cobol filed definition. The program then outputs to a file what is essentially a file which is a list definition which I can later copy and past into a python program. I will eventually expand the program to also output an SQL script to create a SQL file in MySQL The program still need a little work, it does not handle the following items
0
9688
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10490
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10259
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10238
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
6809
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5467
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4145
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3761
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2941
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.