Parsing file in an array with size limit

[yehaimanish]

I am developing an application by which to parse the content from the
access_log and insert it into the database. Since each row is an
different entry, I am using file() to get the contents into an array
and manipulate each row by foreach(...) and insert/update in the
database accordingly.

If the file is small, it works well. If the file is large (say > 5mb),
it generates memory allocation error. However I came to know that the
allowed memory size can be increased, but I want a different approach
for it.

I want like read only first 100 rows (1 to 1000) in first loop and then
1001 to 2000 and so on. I want the the return value in an array.

With file_get_contents(), it is

ac_arr =

string(2575795) "69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET
/img/livechat.gif HTTP/1.1" 200 2450 "http://www.addr.com/faq.htm"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2)
Gecko/20060308 Firefox/1.5.0.2" -
69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET /img/blank.gif
HTTP/1.1" 200 43 "http://www.addr.com/faq.htm" "Mozilla/5.0 (Windows;
U; Windows NT 5.1; en-US; rv:1.8.0.2) Gecko/20060308 Firefox/1.5.0.2" -
69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET /img/tollfree.gif
HTTP/1.1" 200 2332 "http://www.addr.com/faq.htm" "Mozilla/5.0 (Windows;
U; Windows NT 5.1; en-US; rv:1.8.0.2) Gecko/20060308 Firefox/1.5.0.2" -
69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET /img/vers.gif
HTTP/1.1" 200 958 "http://www.addr.com/faq.htm" "Mozilla/5.0 (Windows;
U; Windows NT 5.1; en-US; rv:1.8.0.2) Gecko/20060308 Firefox/1.5.0.2" -


I want like that with file(),

ac_arr =

array(12576) {
[0]=>
string(215) "69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET
/img/livechat.gif HTTP/1.1" 200 2450 "http://www.addr.com/faq.htm"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2)
Gecko/20060308 Firefox/1.5.0.2" -
"
[1]=>
string(210) "69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET
/img/blank.gif HTTP/1.1" 200 43 "http://www.addr.com/faq.htm"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2)
Gecko/20060308 Firefox/1.5.0.2" -
"
[2]=>
string(215) "69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET
/img/tollfree.gif HTTP/1.1" 200 2332 "http://www.addr.com/faq.htm"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2)
Gecko/20060308 Firefox/1.5.0.2" -
"
[3]=>
string(210) "69.254.218.11 - - [27/Apr/2006:10:30:00 -0700] "GET
/img/vers.gif HTTP/1.1" 200 958 "http://www.addr.com/faq.htm"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2)
Gecko/20060308 Firefox/1.5.0.2" -
"

Is there any way to do that (pagination)

Thanks

[Andy Jeffries]

On Wed, 03 May 2006 04:26:21 -0700, ye*********@gmail.com wrote:

I am developing an application by which to parse the content from the
access_log and insert it into the database. Since each row is an different
entry, I am using file() to get the contents into an array and manipulate
each row by foreach(...) and insert/update in the database accordingly.

If the file is small, it works well. If the file is large (say > 5mb), it
generates memory allocation error. However I came to know that the allowed
memory size can be increased, but I want a different approach for it.

Why not use fgets, that returns the data a line at a time? So instead of
using your foreach you use:

$fh = fopen("access_log", "r");
while (!feof($fh)) {
$string = fgets($fh);
...
}
fclose($fh);

With regards to pagination, do that when it's in MySQL using LIMIT.

Cheers,
Andy
--
Andy Jeffries MBCS CITP ZCE | gPHPEdit Lead Developer
http://www.gphpedit.org | PHP editor for Gnome 2
http://www.andyjeffries.co.uk | Personal site and photos

[yehaimanish]

Thanks.

It worked.

[Andy Jeffries]

On Wed, 03 May 2006 04:34:25 -0700, ye*********@gmail.com wrote:

Thanks.

It worked.

Cool, glad to be of help.

Cheers,
Andy

--
Andy Jeffries MBCS CITP ZCE | gPHPEdit Lead Developer
http://www.gphpedit.org | PHP editor for Gnome 2
http://www.andyjeffries.co.uk | Personal site and photos