script execution time too long

[Doug]

I have a script that does a huge database update, which takes a long
time. However, the host I am running on (and I suspect most hosts) are
running in "safe mode." This means I cannot change the maximum execution
time of scripts. So, the script stops right in the middle of doing the
update.

On top of that, I am trying to make code that I can distribute to
others. As of now, the only way that this script can be run is with sys
admin intervention. I would like to make it run without any help from
the system administrator.

Does any one have a good solution for this problem?

-d

[konsu]

i would redesign the database to make it faster and make it so that no admin
intervention would be needed.

"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:Aw*****************@newsread3.news.atl.earthl ink.net...

I have a script that does a huge database update, which takes a long time.
However, the host I am running on (and I suspect most hosts) are running
in "safe mode." This means I cannot change the maximum execution time of
scripts. So, the script stops right in the middle of doing the update.

On top of that, I am trying to make code that I can distribute to others.
As of now, the only way that this script can be run is with sys admin
intervention. I would like to make it run without any help from the
system administrator.

Does any one have a good solution for this problem?

-d

[Doug]

Thanks.... That is an option, but, there is a very large amount of data
involved. Plus the amount of indexes on the data slow down the entry
into the DB.

-d

konsu wrote:

i would redesign the database to make it faster and make it so that no admin
intervention would be needed.

"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:Aw*****************@newsread3.news.atl.earthl ink.net...

I have a script that does a huge database update, which takes a long time.
However, the host I am running on (and I suspect most hosts) are running
in "safe mode." This means I cannot change the maximum execution time of
scripts. So, the script stops right in the middle of doing the update.

On top of that, I am trying to make code that I can distribute to others.
As of now, the only way that this script can be run is with sys admin
intervention. I would like to make it run without any help from the
system administrator.

Does any one have a good solution for this problem?

-d

[konsu]

well, you are talking about a web server processing a request with some html
form data and not being able to finish the prcessing because the database
operation takes too long. the reason why it takes too long is not because
there is a lot of data in the form, otherwise it would choke with all that
data much earlier, maybe even on the client side, so it is not the form, it
is the design of the database. did you try to test it on an empty database?
does it work? if it does then one thing to look at is to segment the
database into pieces. you have to do something with the database because
from what you are saying it looks like it is unmanageable and hence is not
practical to use anymore. it is useless in other words. ;-)

konstantin
"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:EE*****************@newsread3.news.atl.earthl ink.net...

Thanks.... That is an option, but, there is a very large amount of data
involved. Plus the amount of indexes on the data slow down the entry into
the DB.

-d

konsu wrote:

i would redesign the database to make it faster and make it so that no
admin intervention would be needed.

[steve]

"Doug48" wrote:

I have a script that does a huge database update, which takes
a long
time. However, the host I am running on (and I suspect most
hosts) are
running in "safe mode." This means I cannot change the maximum
execution
time of scripts. So, the script stops right in the middle of
doing the
update.

On top of that, I am trying to make code that I can distribute
to
others. As of now, the only way that this script can be run
is with sys
admin intervention. I would like to make it run without any
help from
the system administrator.

Does any one have a good solution for this problem?

-d

Hi Doug,

More info please. Are you running from CLI, or via browser. If
running from browser, then you get around the timeout problem by
having one browser window call another after say, 20 secons, and keep
going until the process is done.

If running from CLI, then I may suggest going with a VPS account which
will get you the flexibility to run the bath job uninterrupted. VPS
can be had these days for $30/month or even less.

You mentioned distributing the script to others? Can you clarify.
Obviously, most users would not be able to run CLI. Then they would
run this process via browser, and the multiple window solution would
do the job (although not reliably, as browser is NOT a good way of
running a long batch job, since it can break easily).

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-script-e...ict178903.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=602510

[Doug]

Let me explain, there is a zipped text file of about 72000 lines (with
zip code data), that must be added to the DB. Once it gets to the
server, it is unzipped, and the zip codes are updated and/or added.

-d

konsu wrote:

well, you are talking about a web server processing a request with some html
form data and not being able to finish the prcessing because the database
operation takes too long. the reason why it takes too long is not because
there is a lot of data in the form, otherwise it would choke with all that
data much earlier, maybe even on the client side, so it is not the form, it
is the design of the database. did you try to test it on an empty database?
does it work? if it does then one thing to look at is to segment the
database into pieces. you have to do something with the database because
from what you are saying it looks like it is unmanageable and hence is not
practical to use anymore. it is useless in other words. ;-)

konstantin
"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:EE*****************@newsread3.news.atl.earthl ink.net...

Thanks.... That is an option, but, there is a very large amount of data
involved. Plus the amount of indexes on the data slow down the entry into
the DB.

-d

konsu wrote:

i would redesign the database to make it faster and make it so that no
admin intervention would be needed.

[Doug]

What's VPS and multiple window solution?

there is a zipped text file of about 72000 lines (with US, us
territories zip code data), that must be added to the DB. Once it gets
to the server, it is unzipped, and the zip codes are updated and/or added.

steve wrote:

"Doug48" wrote:

> I have a script that does a huge database update, which takes
> a long
> time. However, the host I am running on (and I suspect most
> hosts) are
> running in "safe mode." This means I cannot change the maximum
> execution
> time of scripts. So, the script stops right in the middle of
> doing the
> update.
>
> On top of that, I am trying to make code that I can distribute
> to
> others. As of now, the only way that this script can be run
> is with sys
> admin intervention. I would like to make it run without any
> help from
> the system administrator.
>
> Does any one have a good solution for this problem?
>
> -d

Hi Doug,

More info please. Are you running from CLI, or via browser. If
running from browser, then you get around the timeout problem by
having one browser window call another after say, 20 secons, and keep
going until the process is done.

If running from CLI, then I may suggest going with a VPS account which
will get you the flexibility to run the bath job uninterrupted. VPS
can be had these days for $30/month or even less.

You mentioned distributing the script to others? Can you clarify.
Obviously, most users would not be able to run CLI. Then they would
run this process via browser, and the multiple window solution would
do the job (although not reliably, as browser is NOT a good way of
running a long batch job, since it can break easily).

[konsu]

i see ;-)

would you explain why you are processing the file as part of a web request?
(i am still assuming you have a web request ;-)) is there a user who might
want to submit such a file from a web page?

can you create a cron job (assuming you are running unix) or something
similar that would use awk or what else to read the file and update the
database during some night time hours when everyone is sleeping and not
watching? ;-)

konstantin

"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:78****************@newsread2.news.atl.earthli nk.net...

Let me explain, there is a zipped text file of about 72000 lines (with zip
code data), that must be added to the DB. Once it gets to the server, it
is unzipped, and the zip codes are updated and/or added.

-d

[Doug]

Zip codes are updated and changed all the time.

So, PHP scripts run by cron have no time limit?

-d

konsu wrote:

i see ;-)

would you explain why you are processing the file as part of a web request?
(i am still assuming you have a web request ;-)) is there a user who might
want to submit such a file from a web page?

can you create a cron job (assuming you are running unix) or something
similar that would use awk or what else to read the file and update the
database during some night time hours when everyone is sleeping and not
watching? ;-)

konstantin

"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:78****************@newsread2.news.atl.earthli nk.net...

Let me explain, there is a zipped text file of about 72000 lines (with zip
code data), that must be added to the DB. Once it gets to the server, it
is unzipped, and the zip codes are updated and/or added.

-d

[konsu]

> So, PHP scripts run by cron have no time limit?

i do not know about that but i think you have a better control over it. and
if you do not you can always use a non-PHP solution.

[Chung Leong]

"Doug" <do*****@XXXXremovetheXearthlink.net> wrote in message
news:Aw*****************@newsread3.news.atl.earthl ink.net...

I have a script that does a huge database update, which takes a long
time. However, the host I am running on (and I suspect most hosts) are
running in "safe mode." This means I cannot change the maximum execution
time of scripts. So, the script stops right in the middle of doing the
update.

On top of that, I am trying to make code that I can distribute to
others. As of now, the only way that this script can be run is with sys
admin intervention. I would like to make it run without any help from
the system administrator.

Does any one have a good solution for this problem?

-d

Well, break up the process into multiple pieces. The initial request would
process the first X number of lines, then tell the client to refresh
(Javascript). The second request then process the next X number of lines and
so on.

It's a good way to indicate to the user that something is happening too.