473,326 Members | 2,134 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

PHP memory garbage collection

Hi!

I have a script that uses a lot of memory and an ISP that has memory-limit
set to 10 Mb. The script of course doesn't finish running. :(

The program is rather simple - it parses XML into an array and then starts
the loop that inserts data into the database. If I use a very large XML
file then the PHP crashes inside the loop (Fatal error: Allowed memory size
of 10485760 bytes exhausted) - indicating that the problem lies within loop
somewhere.

Can I trace memory consumption of a PHP script so I can determine where the
memory is wasted?

If PHP uses all the memory (and frees nothing) until the end of the script
then that could be the problem - is there any way I can force it to do
garbage collection?

Any comment on how PHP uses memory would be appreciated - I would really
hate to have to break the script into smaller pieces (or change ISP for
that matter :) ).

Thanks!

Anze
Jul 17 '05 #1
11 5767

Anybody? Please? I really need to solve this...

Is there a way to measure current memory consumption?

Regards,

Anze

Anze wrote:
Hi!

I have a script that uses a lot of memory and an ISP that has memory-limit
set to 10 Mb. The script of course doesn't finish running. :(

The program is rather simple - it parses XML into an array and then starts
the loop that inserts data into the database. If I use a very large XML
file then the PHP crashes inside the loop (Fatal error: Allowed memory
size of 10485760 bytes exhausted) - indicating that the problem lies
within loop somewhere.

Can I trace memory consumption of a PHP script so I can determine where
the memory is wasted?

If PHP uses all the memory (and frees nothing) until the end of the script
then that could be the problem - is there any way I can force it to do
garbage collection?

Any comment on how PHP uses memory would be appreciated - I would really
hate to have to break the script into smaller pieces (or change ISP for
that matter :) ).

Thanks!

Anze


Jul 17 '05 #2
Anze wrote:
Anybody? Please? I really need to solve this...

Is there a way to measure current memory consumption?

Regards,

Anze

Anze wrote:
Hi!

I have a script that uses a lot of memory and an ISP that has memory-limit
set to 10 Mb. The script of course doesn't finish running. :(

The program is rather simple - it parses XML into an array and then starts
the loop that inserts data into the database. If I use a very large XML
file then the PHP crashes inside the loop (Fatal error: Allowed memory
size of 10485760 bytes exhausted) - indicating that the problem lies
within loop somewhere.

Can I trace memory consumption of a PHP script so I can determine where
the memory is wasted?

If PHP uses all the memory (and frees nothing) until the end of the script
then that could be the problem - is there any way I can force it to do
garbage collection?

Any comment on how PHP uses memory would be appreciated - I would really
hate to have to break the script into smaller pieces (or change ISP for
that matter :) ).

Thanks!

Anze


I'm already thinking of a solution, my fault...

Look into your script for where you have global variables which you
don't need anymore and then unset them (unset($var)).

If you can't solve it this way, you can contact your ISP to run the
script for you through SSH or something.

I don't think you would run the script often or do you? If you do, I
suggest thinking about the logic and effeciency about your piece of
programming.
Jul 17 '05 #3
Hi!
Look into your script for where you have global variables which you
don't need anymore and then unset them (unset($var)).

If you can't solve it this way, you can contact your ISP to run the
script for you through SSH or something.

I don't think you would run the script often or do you? If you do, I
suggest thinking about the logic and effeciency about your piece of
programming.


First of all, thank you for the answer!

The script will be run only from time to time. The problem is that I already
unset() all the variables that are allocated - at least global ones. All
other are local to functions and should be freed automatically (from what I
understand).

The script is pretty much straightforward - but it does use my CMS as a
backend. That is why I'm trying to see where the memory is being spent. I
am afraid this is only a tip of the iceberg and that other scripts could be
made more effective if I solved this problem.

Just a few minutes ago I found this:
http://www.zend.com/manual/function....-get-usage.php
It is not documented on the PHP site... :(

I will see if I can solve a problem by using this function.

Thank you again!

Regards,

Anze
Jul 17 '05 #4
"Anze" <an******@volja.net> wrote in message
news:OX********************@news.siol.net...
The script is pretty much straightforward - but it does use my CMS as a
backend. That is why I'm trying to see where the memory is being spent. I
am afraid this is only a tip of the iceberg and that other scripts could be made more effective if I solved this problem.


for what is worth I _think_ php uses reference counting for it's memory
management. so maybe you could check if you produce some cyclic references
along the way. Also if you are using php less then version 5 then be aware
of default "copy by value" semantics, which could in some cases leave you
with a quite a few more objects then you expect. And lastly if it is a large
xml file mybe it is a xml parser and iteration over the DOM tree that is
taking over your memory.

If everything fails, maybe you could slice your work to smaller peaces, and
have more runs of the script each doing a smaller chunk of work.

As for finding host that would allow you to use more than 10 megs for php
script, I doubt that you can find one. If you do, he would prpobably have
severe problems with stability and availability of their servers ;)

rush
--
http://www.templatetamer.com/
http://www.folderscavenger.com/
Jul 17 '05 #5
Anze wrote:
Is there a way to measure current memory consumption?

<snip>

http://in.php.net/apd

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

Jul 17 '05 #6
Hi!
for what is worth I _think_ php uses reference counting for it's memory
management. so maybe you could check if you produce some cyclic references
along the way. Also if you are using php less then version 5 then be aware
Yes, I do have them - I just never realized this could be a problem. Thanks
for the tip! But in this script I manually unset() all of the objects with
cyclic references so this shouldn't be a problem, right?
If everything fails, maybe you could slice your work to smaller peaces,
and have more runs of the script each doing a smaller chunk of work.
I would hate to do that - it is the customer that uses this script to upload
some XML with article data and it would be very weird to ask of him to run
the script more than once. :)
As for finding host that would allow you to use more than 10 megs for php
script, I doubt that you can find one. If you do, he would prpobably have
severe problems with stability and availability of their servers ;)


Not really - they could set memory_limit to 10Mb and allow users to change
it via ini_set(). But this provider uses some other memory and time
limiting mechanism (not just memory_limit directive) so there is no way for
me to change it for some scripts.

But there is also an interesting thing I found while debugging. This code
apparently produces memory leaks:

<?php
class A
{
var $as=array();

function A()
{
for ($i=0;$i<9;$i++)
{
$this->as[]='1';
};
}
};

for ($i=0;$i<10;$i++)
{
$a=&new A();
echo "mem: ".memory_get_usage()."<br />\n";
unset($a->as);
unset($a);
};
?>

The output is:
mem: 13984
mem: 14048
mem: 14112
mem: 14176
mem: 14240
mem: 14304
mem: 14368
mem: 14432
mem: 14496
mem: 14560

There is no memory leak if you change this line:
for ($i=0;$i<9;$i++)
to:
for ($i=0;$i<8;$i++)

Is this weird or is it just me? ;)
I'll check bug reports to see if someone else already stumbled across this -
it looks like a bug in PHP.

Regards,

Anze
Jul 17 '05 #7
> http://in.php.net/apd

Not exactly what I had in mind, but great tip anyway - thanks! It looks
promising. :)

Regards,

Anze

Jul 17 '05 #8
> I'll check bug reports to see if someone else already stumbled across this
- it looks like a bug in PHP.


Someone else already reported this bug but developers seem to think this is
normal behaviour:
http://bugs.php.net/bug.php?id=31297

I don't agree with the developers, but then again - if it's only 64 bytes
per object it is not that much. Back to hunting other leaks... ;)

Regards,

Anze
Jul 17 '05 #9

"Anze" <an******@volja.net> wrote in message
news:ef********************@news.siol.net...
Hi!

I have a script that uses a lot of memory and an ISP that has memory-limit
set to 10 Mb. The script of course doesn't finish running. :(

The program is rather simple - it parses XML into an array and then starts
the loop that inserts data into the database. If I use a very large XML
file then the PHP crashes inside the loop (Fatal error: Allowed memory
size
of 10485760 bytes exhausted) - indicating that the problem lies within
loop
somewhere.
There'e your problem right there. You are reading the ENTIRE XML file into
memory before processing it. If you look at the documentation you will find
a method that requires reading the contents of the XML file in little chunks
at a time instead of one big mouthful.

--
Tony Marston

http://www.tonymarston.net
Can I trace memory consumption of a PHP script so I can determine where
the
memory is wasted?

If PHP uses all the memory (and frees nothing) until the end of the script
then that could be the problem - is there any way I can force it to do
garbage collection?

Any comment on how PHP uses memory would be appreciated - I would really
hate to have to break the script into smaller pieces (or change ISP for
that matter :) ).

Thanks!

Anze

Jul 17 '05 #10
> There'e your problem right there. You are reading the ENTIRE XML file into
memory before processing it.


Not really, but thanks anyway. :)

The XML data is small enough and shouldn't pose a problem. It is the
processing that creates many objects and PHP for some reason does not do
garbage collection properly. It could be because of cyclic references - I
don't know. I'm learning how to use APD to test memory usage more
thoroughly. The XML data memory usage is negligible when compared to total
memory usage.

I am using my CMS objects to insert data in DB and this is the easiest way;
I would like to find the reason why PHP doesn't release unused memory
before I dig any deeper.

Thanks again!

Anze
Jul 17 '05 #11
Anze wrote:
There'e your problem right there. You are reading the ENTIRE XML file into memory before processing it.


Not really, but thanks anyway. :)

The XML data is small enough and shouldn't pose a problem.


I agree with Tony, you stated that you only run into this problem, "If
I use a very large XML file". So while the XML parsing may not be the
only problem, it is certainly a significant contributing factor.

If you aren't already using the "XML parser functions"
(http://us2.php.net/xml), then try them. You will find them to be
several times less memory intensive than DOM XML
(http://us2.php.net/domxml).

Jul 17 '05 #12

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Andreas Suurkuusk | last post by:
Hi, I just noticed your post in the "C# memory problem: no end for our problem?" thread. In the post you implied that I do not how the garbage collector works and that I mislead people. Since...
3
by: Ian Taite | last post by:
Hello, I'm exploring why one of my C# .NET apps has "high" memory usage, and whether I can reduce the memory usage. I have an app that wakes up and processes text files into a database...
25
by: Zeng | last post by:
I finally narrowed down my code to this situation, quite a few (not all) of my CMyClass objects got hold up after each run of this function via the simple webpage that shows NumberEd editbox. My...
6
by: Andy | last post by:
Along with many others I've noticed the large amount of memory that can be taken up by the aspnet_wp.exe. I've found that I can better control and limit this memory consumption by including a...
11
by: EDom | last post by:
Hi, I have aspnet_wp.exe with increasing on every postback and not every revisit to any page. Even if I clear session and close the browser it remains in the memory. I have only one connection...
8
by: Adrian | last post by:
Hi I have a JS program that runs localy (under IE6 only) on a PC but it has a memory leak (probably the known MS one!) What applications are there that I could use to look at the memory usage of...
3
by: Jim Land | last post by:
Jack Slocum claims here http://www.jackslocum.com/yui/2006/10/02/3-easy-steps-to-avoid-javascript- memory-leaks/ that "almost every site you visit that uses JavaScript is leaking memory". ...
2
by: roger.dunham | last post by:
I am trying to identify whether a .NET 1.1 application that I have written has a memory leak. I thought I understood how .NET memory management worked, but it appears that there is more to it...
2
by: Jack Jackson | last post by:
On Wed, 11 Jun 2008 07:32:30 +0200, "Fred" <foleide@free.fr.invalid> wrote: If the control will soon be eligible for garbage collection, then no. If the control won't be eligible for garbage...
11
by: dhtml | last post by:
(originally mis-posted on m.p.s.jscript...) I've just closed all windows in Firefox and its using 244MB of memory. I have no idea why. I had GMail open, a page from unicode, the CLJ FAQ. ...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.