hitting the limits

[windandwaves]

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the limits
and I keep getting the 500 error. According to the people from webfarm it
is because my script are too demanding or not closed properly.

I dont believe you have to "close" PHP scripts or even database connections.

The site also accesses a 80 Megabyte database.

Do you know of a way I can find out where the problems are or improving the
performance of the site?

I uses ob_start and ob_end_flush, would that cause problems?

TIA

Nicolaas

[Jerry Stuckle]

windandwaves wrote:

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the limits
and I keep getting the 500 error. According to the people from webfarm it
is because my script are too demanding or not closed properly.

I dont believe you have to "close" PHP scripts or even database connections.

The site also accesses a 80 Megabyte database.

Do you know of a way I can find out where the problems are or improving the
performance of the site?

I uses ob_start and ob_end_flush, would that cause problems?

TIA

Nicolaas

Well, ob_start and ob_end_flush require system resources; the bigger the page
the more resources it requires.

You might not be using many resources with it. You may be using a lot. It all
depends on the site and what you're using it for.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================

[Rik]

windandwaves wrote:

I am managing a site, www.friars.co.nz that seems to be hitting the
limits and I keep getting the 500 error. According to the people
from webfarm it is because my script are too demanding
I'm no server admin, but maybe you can ask them what the most demanding
requests are?
or not closed
properly.

I dont believe you have to "close" PHP scripts
Not to my knowledge, no.
or even database
connections.
Normally no.
If your resources are stretched, it may be worth it to check wether your
script opens once (and only once) a connection to the database per request,
extracts all needed data, and then closes the connection, after which is
continues further processing. If your website is that popular, maybe there
is something to gain from using a persistent database connection, that
highly depends on your used scripts and connections.

I might be talking out of my ass here, I've unfortunately never had the
problem of being so popular :-).
The site also accesses a 80 Megabyte database.
Phah! That's huge compared to what I'm used to.
I assume you've normalized the database?
Created proper indexes for faster selecting?
Do you know of a way I can find out where the problems are or
improving the performance of the site?
To check how much time (and probably resource) your script takes, loop over
it a couple of times on a local server (somewhere in the 100 or more), and
on specific places in your code add the following:

At the start:
$start = microtime(true);

At key locations in the script you want to know the time taken.
$end = microtime(true);
$time_taken[$some_specific_name] += $end-$start;
$start = $end;

print_r($time_taken) after a 100 or so loops will make it clear to you which
portions of your code take the most time, and check those specific portions
wether they can be done more effective.

If PHP < 5 use www.php.net's suggestion: create a function:
function microtime_float()
{
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);
}
and replace microtime(true) with microtime_float().
I uses ob_start and ob_end_flush, would that cause problems?

It increases the use of resources. When stretched it might be a problem,
normally it won't. I wouldn't think it's main cause of the problem, but it
will add to it. Is there a specific reason you NEED ob functions? If not:
don't use them.
To decrease server-load you could think about a cache system: review the
dependancies of databasefields for each of your pages, create appropriate
timestamps with an index in the database if they don't exist already, and
cache the pages locally as html with a date/time. On a request, check wether
there is a timestamp in the database higher than of your cached html If not,
serve the cached file, if so, create new cachefile and serve that. Creating
of cachefiles should offcourse take place automatically on an update of a
certain database field, but that might be even more work.

I hope I've been of some help,
--
Rik Wasmus

[Shaun]

On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
<wi*********@coldmail.com> wrote:

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the limits
and I keep getting the 500 error. According to the people from webfarm it
is because my script are too demanding or not closed properly.

I dont believe you have to "close" PHP scripts or even database connections.
Ideally you shouldn't have to do either. PHP scripts automatically
close all resources upon termination, unless you've explicitly asked
them to do otherwise.

I've rarely encountered a 500 error using PHP, typically it crops up
with Perl scripts. In most situations, PHP will throw its own errors,
which are descriptive enough to trace the source of the problem.
(Granted, I just posted a few moments ago about an exception to this
case...)
The site also accesses a 80 Megabyte database.
Database as in...? MySQL? How much traffic are you getting? Are you
using persistent database connections, or normal ones?
Do you know of a way I can find out where the problems are or improving the
performance of the site?
I'm not familiar with webfarm; with any luck, they give you a PHP
error log, or they send PHP's errors into Apache's error log. Whatever
error logs are available to you, check them religiously.
I uses ob_start and ob_end_flush, would that cause problems?

Jerry did that one justice in his response.

hth

-
Remove mypants to email.
<http://www.shaunc.com/>

[windandwaves]

Jerry Stuckle wrote:

windandwaves wrote:

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the
limits and I keep getting the 500 error. According to the people
from webfarm it is because my script are too demanding or not closed
properly. I dont believe you have to "close" PHP scripts or even database
connections. The site also accesses a 80 Megabyte database.

Do you know of a way I can find out where the problems are or
improving the performance of the site?

I uses ob_start and ob_end_flush, would that cause problems?

TIA

Nicolaas

Well, ob_start and ob_end_flush require system resources; the bigger
the page the more resources it requires.

You might not be using many resources with it. You may be using a
lot. It all depends on the site and what you're using it for.

Hmm, yes, the good thin with ob_end is that I can resize the actual HMTL to
be very small, meaning that what we send is small - at least.

[windandwaves]

Rik wrote:
....

To decrease server-load you could think about a cache system: review
the dependancies of databasefields for each of your pages, create
appropriate timestamps with an index in the database if they don't
exist already, and cache the pages locally as html with a date/time.
On a request, check wether there is a timestamp in the database
higher than of your cached html If not, serve the cached file, if so,
create new cachefile and serve that. Creating of cachefiles should
offcourse take place automatically on an update of a certain database
field, but that might be even more work.

Rik, that is a cool idea, would you just place the html in a database or is
there a special chache where pages are kept?

[windandwaves]

Shaun wrote:

On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
<wi*********@coldmail.com> wrote:

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the
limits and I keep getting the 500 error. According to the people
from webfarm it is because my script are too demanding or not closed
properly.

Hmmm, yes, we have only about 10,000 unique visitors per month and a
relatively straight forward MySql database. However, we have shared
hosting, I think that is where the problem lies....

Thanks for your reply
Nicolaas

[Rik]

windandwaves wrote:

Rik wrote:
...

To decrease server-load you could think about a cache system: review
the dependancies of databasefields for each of your pages, create
appropriate timestamps with an index in the database if they don't
exist already, and cache the pages locally as html with a date/time.
On a request, check wether there is a timestamp in the database
higher than of your cached html If not, serve the cached file, if so,
create new cachefile and serve that. Creating of cachefiles should
offcourse take place automatically on an update of a certain database
field, but that might be even more work.

Rik, that is a cool idea, would you just place the html in a database
or is there a special chache where pages are kept?

An HTML page can perfectly be kept in the database, makes it even easier to
check wether a page has to be "rewritten" or not, possibly in 1 simple query
returning 0 or 1.

And if some items in a webpage aren't cachable, you could always try to
cache certain portions of code.

Grtz,
--
Rik Wasmus

[Jerry Stuckle]

windandwaves wrote:

Jerry Stuckle wrote:

windandwaves wrote:

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the
limits and I keep getting the 500 error. According to the people
from webfarm it is because my script are too demanding or not closed
properly. I dont believe you have to "close" PHP scripts or even database
connections. The site also accesses a 80 Megabyte database.

Do you know of a way I can find out where the problems are or
improving the performance of the site?

I uses ob_start and ob_end_flush, would that cause problems?

TIA

Nicolaas

Well, ob_start and ob_end_flush require system resources; the bigger
the page the more resources it requires.

You might not be using many resources with it. You may be using a
lot. It all depends on the site and what you're using it for.

Hmm, yes, the good thin with ob_end is that I can resize the actual HMTL to
be very small, meaning that what we send is small - at least.

Why even us it? I find very seldom do I need it. And it does add overhead.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================

[Jerry Stuckle]

windandwaves wrote:

Shaun wrote:

On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
<wi*********@coldmail.com> wrote:

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the
limits and I keep getting the 500 error. According to the people
from webfarm it is because my script are too demanding or not closed
properly.

Hmmm, yes, we have only about 10,000 unique visitors per month and a
relatively straight forward MySql database. However, we have shared
hosting, I think that is where the problem lies....

Thanks for your reply

Nicolaas

10K visitors a month and an 80Mb database are nothing. Any shared host should
be able to handle that, also.

If you think it's a problem, talk to your hosting company. Maybe the server
you're on is overloaded and they will move you to a less heavily loaded server.

I'm not familiar with webfarm - but they may be right, also. For instance, if
you don't close your mysql connection, it will be closed eventually. But the
connection will hang around until the garbage collector gets around to cleaning
things up.

And look at how efficient your code is. I've seen some pretty good code - but
I've seen some very inefficient code out there, also. And the inefficient code
will require a lot more resources.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================

[windandwaves]

Jerry Stuckle wrote:
.....

Why even us it? I find very seldom do I need it. And it does add
overhead.
I use it on all my sites because in that way, I can perfectly indent my code
but take out the 100s or even 1000s of spaces/tabs.

For all my sites, I make a login and when the person is logged in (i.e. me
the administrator), the page is produced with spaces so that I can analyse
the code.

I like writing xhtml strict with real simple xhtml that is compressed (i.e.
without spaces) so that the actual pages are super small. I do this to
protest against the dreamweavers out there, who create html that is often
over 100kb and just pollutes the superhighway with endless &nbsp; <td><img
src="spacer.gif">etc......

HTMS
Nicolaas

[windandwaves]

Jerry Stuckle wrote:

windandwaves wrote:

Shaun wrote:

On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
<wi*********@coldmail.com> wrote:
Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the
limits and I keep getting the 500 error. According to the people
from webfarm it is because my script are too demanding or not
closed properly.
Hmmm, yes, we have only about 10,000 unique visitors per month and a
relatively straight forward MySql database. However, we have shared
hosting, I think that is where the problem lies....

Thanks for your reply

Nicolaas

10K visitors a month and an 80Mb database are nothing. Any shared
host should be able to handle that, also.

If you think it's a problem, talk to your hosting company. Maybe the
server you're on is overloaded and they will move you to a less
heavily loaded server.
I'm not familiar with webfarm - but they may be right, also. For
instance, if you don't close your mysql connection, it will be closed
eventually. But the connection will hang around until the garbage
collector gets around to cleaning things up.

I will do a close database connection at the end of each script and see if
it makes a difference.
And look at how efficient your code is. I've seen some pretty good
code - but I've seen some very inefficient code out there, also. And
the inefficient code will require a lot more resources.
It is so hard for me to judge if the code is efficient. Really - who knows.
I know what efficient code is, but there are just so many variables. Even
making changes tot the php.ini could be huge.

Each page loads about 100Kb of libaries and functions... (probably 50 - 100
functions (small ones) in total)... Does that make a difference?
I have a couple of tables in the database with over 300,000 (small) rows)...
Does that make a difference?
I use a lot of session variables (i.e. I keep a log of each page visited for
each person), but all of them are stored in highly optimised Mysql tables
(e.g.

person ID, page ID, time
person ID, page ID, time
person ID, page ID, time
person ID, page ID, time

In the end, because it is such muddy water, it is probably cheaper to pay
for a dedicated server than to spend hours optimising your code. I can do
fast development and write code in such a way that it is easy to maintain.

Thanks for your reply. Much appreciated.
Nicolaas

[Jerry Stuckle]

windandwaves wrote:

Jerry Stuckle wrote:
....

Why even us it? I find very seldom do I need it. And it does add
overhead.

I use it on all my sites because in that way, I can perfectly indent my code
but take out the 100s or even 1000s of spaces/tabs.

For all my sites, I make a login and when the person is logged in (i.e. me
the administrator), the page is produced with spaces so that I can analyse
the code.

I like writing xhtml strict with real simple xhtml that is compressed (i.e.
without spaces) so that the actual pages are super small. I do this to
protest against the dreamweavers out there, who create html that is often
over 100kb and just pollutes the superhighway with endless &nbsp; <td><img
src="spacer.gif">etc......

HTMS

Nicolaas

You can write perfectly good xml or html without obstart(). The two have
nothing to do with each other.

But compressing a page then expanding it just to make the code more readable
doesn't make sense. Just expand it in the file and serve it statically.
There's much less overhead.

I think the only time I've really needed obstart() is when I wanted to wrap
phpinfo() in another page. In that case I needed to get the output, parse it,
getting rid of the extra tags, then print it.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================

[Rik]

windandwaves wrote:

It is so hard for me to judge if the code is efficient. Really - who
knows. I know what efficient code is, but there are just so many
variables. Even making changes tot the php.ini could be huge.
Yup, but previous suggestion with microtime() would give you an indication
where the most time is taken.
Each page loads about 100Kb of libaries and functions... (probably 50
- 100 functions (small ones) in total)... Does that make a difference?
Yes, probably not a lot, but why?
I'm having great difficulties imagining a a website where that much
libraries and functions are needed. Maybe it's time to include only files
that are actually needed?

It may be worth it to check the following url:
http://nl2.php.net/manual/en/language.oop5.autoload.php
I have a couple of tables in the database with over 300,000 (small)
rows)... Does that make a difference?

Having a database doesn't make a differnce.
Exactly HOW you query the database does.

Normalize and create proper indexes.-

Grtz,
--
Rik Wasmus

[Jerry Stuckle]

windandwaves wrote:

Jerry Stuckle wrote:

windandwaves wrote:

Shaun wrote:
On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
<wi*********@coldmail.com> wrote:

>Hi Folk
>
>I am managing a site, www.friars.co.nz that seems to be hitting the
>limits and I keep getting the 500 error. According to the people
>from webfarm it is because my script are too demanding or not
>closed properly.

Hmmm, yes, we have only about 10,000 unique visitors per month and a
relatively straight forward MySql database. However, we have shared
hosting, I think that is where the problem lies....

Thanks for your reply

Nicolaas

10K visitors a month and an 80Mb database are nothing. Any shared
host should be able to handle that, also.

If you think it's a problem, talk to your hosting company. Maybe the
server you're on is overloaded and they will move you to a less
heavily loaded server.
I'm not familiar with webfarm - but they may be right, also. For
instance, if you don't close your mysql connection, it will be closed
eventually. But the connection will hang around until the garbage
collector gets around to cleaning things up.

I will do a close database connection at the end of each script and see if
it makes a difference.

And look at how efficient your code is. I've seen some pretty good
code - but I've seen some very inefficient code out there, also. And
the inefficient code will require a lot more resources.

It is so hard for me to judge if the code is efficient. Really - who knows.
I know what efficient code is, but there are just so many variables. Even
making changes tot the php.ini could be huge.

Each page loads about 100Kb of libaries and functions... (probably 50 - 100
functions (small ones) in total)... Does that make a difference?
I have a couple of tables in the database with over 300,000 (small) rows)...
Does that make a difference?
I use a lot of session variables (i.e. I keep a log of each page visited for
each person), but all of them are stored in highly optimised Mysql tables
(e.g.

person ID, page ID, time
person ID, page ID, time
person ID, page ID, time
person ID, page ID, time

In the end, because it is such muddy water, it is probably cheaper to pay
for a dedicated server than to spend hours optimising your code. I can do
fast development and write code in such a way that it is easy to maintain.

Thanks for your reply. Much appreciated.

Nicolaas

Well, if every page is loading unnecessary functions, then yes, that will add
overhead. PHP has to parse all that code. I tend to break my functions in to
groups, and only load those which are necessary for the page.

The database itself isn't a problem. However, your access to the database could
be inefficient. For instance, you may be doing full table scans when the
appropriate index could allow an index scan. But that is all very dependent on
your the RDB you're using, database layout, the actual operations you're
performing on it and about a dozen other variables.

I wouldn't think session variables would have a huge effect on it, but it's also
possible. However, I would also look at what you're doing. Can you get
basically the same information from your Apache log?

Yes, it will take time to make it more efficient. But I think it would be worth
it. And a dedicated server has its own additional overhead for maintenance.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================

[windandwaves]

Jerry Stuckle wrote:

windandwaves wrote:

Jerry Stuckle wrote:
....

Why even us it? I find very seldom do I need it. And it does add
overhead.

I use it on all my sites because in that way, I can perfectly indent
my code but take out the 100s or even 1000s of spaces/tabs.

For all my sites, I make a login and when the person is logged in
(i.e. me the administrator), the page is produced with spaces so
that I can analyse the code.

I like writing xhtml strict with real simple xhtml that is
compressed (i.e. without spaces) so that the actual pages are super
small. I do this to protest against the dreamweavers out there, who
create html that is often over 100kb and just pollutes the
superhighway with endless &nbsp; <td><img src="spacer.gif">etc......

HTMS

Nicolaas

You can write perfectly good xml or html without obstart(). The two
have nothing to do with each other.

But compressing a page then expanding it just to make the code more
readable doesn't make sense. Just expand it in the file and serve it
statically. There's much less overhead.

What I do is for any user that is not logged-in, I compress the page getting
rid of the 1000s of spaces and tabs. For the select few that log-in, I do
not compress it so I can quickly spot errors in the code.
I think the only time I've really needed obstart() is when I wanted
to wrap phpinfo() in another page. In that case I needed to get the
output, parse it, getting rid of the extra tags, then print it.

I just like ditching all the html at once. Bang, 3Kb of perfectly clean
(x)html. The css, javascript and images are bulkier, but these are often
cached. Hmmm, you are probably right though, I would involved a LOT of
changes :-)

[windandwaves]

Rik wrote:

windandwaves wrote:

It is so hard for me to judge if the code is efficient. Really - who
knows. I know what efficient code is, but there are just so many
variables. Even making changes tot the php.ini could be huge.
Yup, but previous suggestion with microtime() would give you an
indication where the most time is taken.

Yes, I may try that, but i found that pages load superfast one second and
almost not at all the next. I did some database work last night with a
script running through an entire table of 20,000 records one by one. I did
an echo for each line and it was really interesting to see that sometimes
the entire screen filled with echos within a split second, while the next
moment, each line would literally take one second.

Each page loads about 100Kb of libaries and functions... (probably 50
- 100 functions (small ones) in total)... Does that make a
difference?
Yes, probably not a lot, but why?
I'm having great difficulties imagining a a website where that much
libraries and functions are needed. Maybe it's time to include only
files that are actually needed?

Yes, that is for sure, but sorting out exactly what is needed for each page
take expensive development time.

It may be worth it to check the following url:
http://nl2.php.net/manual/en/language.oop5.autoload.php

I have a couple of tables in the database with over 300,000 (small)
rows)... Does that make a difference?
Having a database doesn't make a differnce.
Exactly HOW you query the database does.

Lol, yes, I guess you are right. I have added lots of indexes so querying
should be pretty fast, but adding may take longer....

I have one unique index that compromises five fields, would that matter?

So many questions, so few clues
Normalize and create proper indexes.-

Grtz,

[Jerry Stuckle]

windandwaves wrote:

Jerry Stuckle wrote:

windandwaves wrote:

Jerry Stuckle wrote:
....
Why even us it? I find very seldom do I need it. And it does add
overhead.
I use it on all my sites because in that way, I can perfectly indent
my code but take out the 100s or even 1000s of spaces/tabs.

For all my sites, I make a login and when the person is logged in
(i.e. me the administrator), the page is produced with spaces so
that I can analyse the code.

I like writing xhtml strict with real simple xhtml that is
compressed (i.e. without spaces) so that the actual pages are super
small. I do this to protest against the dreamweavers out there, who
create html that is often over 100kb and just pollutes the
superhighway with endless &nbsp; <td><img src="spacer.gif">etc......

HTMS

Nicolaas

You can write perfectly good xml or html without obstart(). The two
have nothing to do with each other.

But compressing a page then expanding it just to make the code more
readable doesn't make sense. Just expand it in the file and serve it
statically. There's much less overhead.

What I do is for any user that is not logged-in, I compress the page getting
rid of the 1000s of spaces and tabs. For the select few that log-in, I do
not compress it so I can quickly spot errors in the code.

I think the only time I've really needed obstart() is when I wanted
to wrap phpinfo() in another page. In that case I needed to get the
output, parse it, getting rid of the extra tags, then print it.

I just like ditching all the html at once. Bang, 3Kb of perfectly clean
(x)html. The css, javascript and images are bulkier, but these are often
cached. Hmmm, you are probably right though, I would involved a LOT of
changes :-)

And what happens if you have a problem with your compression?

I'm now beginning to see why your provider is suspecting your scripts are a
problem.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================

[windandwaves]

Jerry Stuckle wrote:
....snip....

And what happens if you have a problem with your compression?

I'm now beginning to see why your provider is suspecting your scripts
are a problem.

when I say compression, I just mean taking out the extra spaces, nothing
else. that is all I do. I load all the script outputs to the buffer and
then remove any double spaces and tabs. The funny thing is that in the last
day, the server runs a lot, lot faster.... With the same amount of visitors.
I am still

[Jeff North]

On Sun, 30 Apr 2006 13:44:43 +1200, in comp.lang.php "windandwaves"
<wi*********@coldmail.com>
<uE*****************@news.xtra.co.nz> wrote:

| Rik wrote:
| > windandwaves wrote:
| >> It is so hard for me to judge if the code is efficient. Really - who
| >> knows. I know what efficient code is, but there are just so many
| >> variables. Even making changes tot the php.ini could be huge.
| >
| > Yup, but previous suggestion with microtime() would give you an
| > indication where the most time is taken.
|
| Yes, I may try that, but i found that pages load superfast one second and
| almost not at all the next. I did some database work last night with a
| script running through an entire table of 20,000 records one by one. I did
| an echo for each line and it was really interesting to see that sometimes
| the entire screen filled with echos within a split second, while the next
| moment, each line would literally take one second.

When was the last time you 'defragged' your tables?
Optimize table <tablename>
Repair table <tablename> EXTENDED USE_FRM
---------------------------------------------------------------
I often wish that email had never been invented, but there’s
just no way I can get rid of it. So, day after day, several times
a day, I dutifully delete 99% of the emails I receive, and when
I’m not able to get at my email for a few days, I’ll leave the
machine at home running to pick it up every 10 minutes so I don’t
overflow some capacity somewhere, and just the other day I caught
myself wondering who will clean out my Inbox after I’m dead.

Charles Petzold. October 20, 2005
---------------------------------------------------------------