Memory limit reached during CLI query, or what?

Running a CLI script that fopen's a file, parses the lines into an
array, checks the array entries against a few regular expression
qualifiers (i.e. !eregi("bot",$entry)) and dump the good entries into
a MySQL db.

The fopen'ed file is a 410MB text file with log entries formatted like
a .CSV file ("" text qualifiers and comma separators).

Running the script from a web browser in Windows98: Works fine, except
it stops about 1/10th of the way through (@~40MB).

Running the script via CLI on the RH9 server: Works fine, except I get
"Terminated" about 3/4rs of the way through (@~300MB).

Am I running into a memory useage issue or what? There's no
distinction between the last entry run successfully and the one after
that, in the log file dump, and everything goes into the db
fine...until the script terminates.

The Windows98 machine is running 512MB/PC133 DRAM with a 120GB HDD,
connected via a full T1 to the RH9 server with the MySQL db.

The RH9 server is a production server (I know...but I'm running out of
options on where to process this behemoth text file!) running 1GB RAM
and twin 60GB HDDs.

I will probably take what I have, trim the file to the remaining
entries, and append to the db, however I would like to understand why
the script is cutting out.

Thanks in advance for any illumination!

re: Memory limit reached during CLI query, or what?

On 6 Aug 2004 13:01:04 -0700
stupidscript@hotmail.com (James Butler) wrote:
[color=blue]
> Running a CLI script that fopen's a file, parses the lines into an
> array, checks the array entries against a few regular expression
> qualifiers (i.e. !eregi("bot",$entry)) and dump the good entries into
> a MySQL db.
>
> The fopen'ed file is a 410MB text file with log entries formatted like
> a .CSV file ("" text qualifiers and comma separators).
>
> Running the script from a web browser in Windows98: Works fine, except
> it stops about 1/10th of the way through (@~40MB).
>[/color]

Try setting max execution time a little higher in the script, I could
imagine that is what blocks it. (I think the default is 30 seconds.)
E.g.:
set_time_limit(0);
to make it run until the script is done.

[snip]

Don't know about the CLI issue though. :(

Best regards,
Madsen

--
Anders K. Madsen --- http://lillesvin.linux.dk

"There are 10 types of people in the world.
Those who understand binary - and those who don't."


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFBE/Q0lNHJe/JASHcRAnm6AJ9P/uF+K7mOk2ARMJgI4HB4b2cQjQCggYXE
2+d8N8F3vzOn1PCcAkXHe5Y=
=+GLO
-----END PGP SIGNATURE-----

re: Memory limit reached during CLI query, or what?

"James Butler" <stupidscript@hotmail.com> wrote in message
news:5e377339.0408061201.7d116953@posting.google.c om...[color=blue]
> Running a CLI script that fopen's a file, parses the lines into an
> array, checks the array entries against a few regular expression
> qualifiers (i.e. !eregi("bot",$entry)) and dump the good entries into
> a MySQL db.
>
> The fopen'ed file is a 410MB text file with log entries formatted like
> a .CSV file ("" text qualifiers and comma separators).
>
> Running the script from a web browser in Windows98: Works fine, except
> it stops about 1/10th of the way through (@~40MB).
>
> Running the script via CLI on the RH9 server: Works fine, except I get
> "Terminated" about 3/4rs of the way through (@~300MB).
>
> Am I running into a memory useage issue or what? There's no
> distinction between the last entry run successfully and the one after
> that, in the log file dump, and everything goes into the db
> fine...until the script terminates.
>
> The Windows98 machine is running 512MB/PC133 DRAM with a 120GB HDD,
> connected via a full T1 to the RH9 server with the MySQL db.
>
> The RH9 server is a production server (I know...but I'm running out of
> options on where to process this behemoth text file!) running 1GB RAM
> and twin 60GB HDDs.
>
> I will probably take what I have, trim the file to the remaining
> entries, and append to the db, however I would like to understand why
> the script is cutting out.
>
> Thanks in advance for any illumination![/color]

Very well could be a memory issue. PHP4 is rather lousy when it comes to
memory management. In my experience, stability issue start to crop up once
the amount the data processed exceeds 100 MB. Hard to tell without seeing
some code though.

(I'm going on the assumption that you've turned off time limit. Since you're
doing a database insert, I don't think the code would have gone half as far
if it were on.)

re: Memory limit reached during CLI query, or what?

"James Butler" wrote:[color=blue]
> Running a CLI script that fopen’s a file, parses the lines into
> an
> array, checks the array entries against a few regular expression
> qualifiers (i.e. !eregi("bot",$entry)) and dump the good entries[/color]
into[color=blue]
> a MySQL db.
>
> The fopen’ed file is a 410MB text file with log entries
> formatted like
> a .CSV file ("" text qualifiers and comma separators).
>
> Running the script from a web browser in Windows98: Works fine,[/color]
except[color=blue]
> it stops about 1/10th of the way through (@~40MB).
>
> Running the script via CLI on the RH9 server: Works fine, except I[/color]
get[color=blue]
> "Terminated" about 3/4rs of the way through (@~300MB).
>
> Am I running into a memory useage issue or what? There’s no
> distinction between the last entry run successfully and the one[/color]
after[color=blue]
> that, in the log file dump, and everything goes into the db
> fine...until the script terminates.
>
> The Windows98 machine is running 512MB/PC133 DRAM with a 120GB HDD,
> connected via a full T1 to the RH9 server with the MySQL db.
>
> The RH9 server is a production server (I know...but I’m running
> out of
> options on where to process this behemoth text file!) running 1GB[/color]
RAM[color=blue]
> and twin 60GB HDDs.
>
> I will probably take what I have, trim the file to the remaining
> entries, and append to the db, however I would like to understand[/color]
why[color=blue]
> the script is cutting out.
>
> Thanks in advance for any illumination![/color]

I believe you can use fread and read it chunk at a time (and discard
what you have just read).

"Watch out when you use readfile to read big files ! Reading a file
of 6 meg will result in php using 6 megs of memory ! Php might stop
your script if you cross the memory limit. You’re better of using
fread when reading big files " from
http://ca3.php.net/manual/en/function.readfile.php

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-Memory-l...ict137374.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=459616

re: Memory limit reached during CLI query, or what?

Thank you all for your thoughts. I believe you are all correct that it
is, in fact, a memory issue. I have decided to particlize the
process...using chunk files @~10MB each and processing them one at a
time, not as a batch. Yeah, that's around 40 iterations of the
process, but at least I can get the data into the database, finally.

I appreciate all of your help.

re: Memory limit reached during CLI query, or what?

"James Butler" wrote:[color=blue]
> Thank you all for your thoughts. I believe you are all correct that[/color]
it[color=blue]
> is, in fact, a memory issue. I have decided to particlize the
> process...using chunk files @~10MB each and processing them one at[/color]
a[color=blue]
> time, not as a batch. Yeah, that’s around 40 iterations of the
> process, but at least I can get the data into the database,[/color]
finally.[color=blue]
>
> I appreciate all of your help.[/color]

James, did you see my comment? I don’t think you have to do all that.
Just use fread.

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-Memory-l...ict137374.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=461323