script execution time too long

I have a script that does a huge database update, which takes a long
time. However, the host I am running on (and I suspect most hosts) are
running in "safe mode." This means I cannot change the maximum execution
time of scripts. So, the script stops right in the middle of doing the
update.

On top of that, I am trying to make code that I can distribute to
others. As of now, the only way that this script can be run is with sys
admin intervention. I would like to make it run without any help from
the system administrator.

Does any one have a good solution for this problem?

-d

re: script execution time too long

i would redesign the database to make it faster and make it so that no admin
intervention would be needed.

"Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
news:Aw9vd.1969$Yj4.1819@newsread3.news.atl.earthl ink.net...[color=blue]
>
>
> I have a script that does a huge database update, which takes a long time.
> However, the host I am running on (and I suspect most hosts) are running
> in "safe mode." This means I cannot change the maximum execution time of
> scripts. So, the script stops right in the middle of doing the update.
>
> On top of that, I am trying to make code that I can distribute to others.
> As of now, the only way that this script can be run is with sys admin
> intervention. I would like to make it run without any help from the
> system administrator.
>
> Does any one have a good solution for this problem?
>
> -d[/color]

re: script execution time too long

Thanks.... That is an option, but, there is a very large amount of data
involved. Plus the amount of indexes on the data slow down the entry
into the DB.

-d

konsu wrote:[color=blue]
> i would redesign the database to make it faster and make it so that no admin
> intervention would be needed.
>
> "Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
> news:Aw9vd.1969$Yj4.1819@newsread3.news.atl.earthl ink.net...
>[color=green]
>>
>>I have a script that does a huge database update, which takes a long time.
>>However, the host I am running on (and I suspect most hosts) are running
>>in "safe mode." This means I cannot change the maximum execution time of
>>scripts. So, the script stops right in the middle of doing the update.
>>
>>On top of that, I am trying to make code that I can distribute to others.
>>As of now, the only way that this script can be run is with sys admin
>>intervention. I would like to make it run without any help from the
>>system administrator.
>>
>>Does any one have a good solution for this problem?
>>
>>-d[/color]
>
>
>[/color]

re: script execution time too long

well, you are talking about a web server processing a request with some html
form data and not being able to finish the prcessing because the database
operation takes too long. the reason why it takes too long is not because
there is a lot of data in the form, otherwise it would choke with all that
data much earlier, maybe even on the client side, so it is not the form, it
is the design of the database. did you try to test it on an empty database?
does it work? if it does then one thing to look at is to segment the
database into pieces. you have to do something with the database because
from what you are saying it looks like it is unmanageable and hence is not
practical to use anymore. it is useless in other words. ;-)

konstantin


"Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
news:EEovd.4016$Yj4.3039@newsread3.news.atl.earthl ink.net...[color=blue]
>
> Thanks.... That is an option, but, there is a very large amount of data
> involved. Plus the amount of indexes on the data slow down the entry into
> the DB.
>
> -d
>
> konsu wrote:[color=green]
>> i would redesign the database to make it faster and make it so that no
>> admin intervention would be needed.
>>[/color][/color]

re: script execution time too long

"Doug48" wrote:[color=blue]
> I have a script that does a huge database update, which takes
> a long
> time. However, the host I am running on (and I suspect most
> hosts) are
> running in "safe mode." This means I cannot change the maximum
> execution
> time of scripts. So, the script stops right in the middle of
> doing the
> update.
>
> On top of that, I am trying to make code that I can distribute
> to
> others. As of now, the only way that this script can be run
> is with sys
> admin intervention. I would like to make it run without any
> help from
> the system administrator.
>
> Does any one have a good solution for this problem?
>
> -d[/color]

Hi Doug,

More info please. Are you running from CLI, or via browser. If
running from browser, then you get around the timeout problem by
having one browser window call another after say, 20 secons, and keep
going until the process is done.

If running from CLI, then I may suggest going with a VPS account which
will get you the flexibility to run the bath job uninterrupted. VPS
can be had these days for $30/month or even less.

You mentioned distributing the script to others? Can you clarify.
Obviously, most users would not be able to run CLI. Then they would
run this process via browser, and the multiple window solution would
do the job (although not reliably, as browser is NOT a good way of
running a long batch job, since it can break easily).

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-script-e...ict178903.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=602510

re: script execution time too long

Let me explain, there is a zipped text file of about 72000 lines (with
zip code data), that must be added to the DB. Once it gets to the
server, it is unzipped, and the zip codes are updated and/or added.

-d

konsu wrote:
[color=blue]
> well, you are talking about a web server processing a request with some html
> form data and not being able to finish the prcessing because the database
> operation takes too long. the reason why it takes too long is not because
> there is a lot of data in the form, otherwise it would choke with all that
> data much earlier, maybe even on the client side, so it is not the form, it
> is the design of the database. did you try to test it on an empty database?
> does it work? if it does then one thing to look at is to segment the
> database into pieces. you have to do something with the database because
> from what you are saying it looks like it is unmanageable and hence is not
> practical to use anymore. it is useless in other words. ;-)
>
> konstantin
>
>
> "Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
> news:EEovd.4016$Yj4.3039@newsread3.news.atl.earthl ink.net...
>[color=green]
>>Thanks.... That is an option, but, there is a very large amount of data
>>involved. Plus the amount of indexes on the data slow down the entry into
>>the DB.
>>
>>-d
>>
>>konsu wrote:
>>[color=darkred]
>>>i would redesign the database to make it faster and make it so that no
>>>admin intervention would be needed.
>>>[/color][/color]
>
>
>[/color]

re: script execution time too long

What's VPS and multiple window solution?

there is a zipped text file of about 72000 lines (with US, us
territories zip code data), that must be added to the DB. Once it gets
to the server, it is unzipped, and the zip codes are updated and/or added.

steve wrote:
[color=blue]
> "Doug48" wrote:[color=green]
> > I have a script that does a huge database update, which takes
> > a long
> > time. However, the host I am running on (and I suspect most
> > hosts) are
> > running in "safe mode." This means I cannot change the maximum
> > execution
> > time of scripts. So, the script stops right in the middle of
> > doing the
> > update.
> >
> > On top of that, I am trying to make code that I can distribute
> > to
> > others. As of now, the only way that this script can be run
> > is with sys
> > admin intervention. I would like to make it run without any
> > help from
> > the system administrator.
> >
> > Does any one have a good solution for this problem?
> >
> > -d[/color]
>
> Hi Doug,
>
> More info please. Are you running from CLI, or via browser. If
> running from browser, then you get around the timeout problem by
> having one browser window call another after say, 20 secons, and keep
> going until the process is done.
>
> If running from CLI, then I may suggest going with a VPS account which
> will get you the flexibility to run the bath job uninterrupted. VPS
> can be had these days for $30/month or even less.
>
> You mentioned distributing the script to others? Can you clarify.
> Obviously, most users would not be able to run CLI. Then they would
> run this process via browser, and the multiple window solution would
> do the job (although not reliably, as browser is NOT a good way of
> running a long batch job, since it can break easily).
>[/color]

re: script execution time too long

i see ;-)

would you explain why you are processing the file as part of a web request?
(i am still assuming you have a web request ;-)) is there a user who might
want to submit such a file from a web page?

can you create a cron job (assuming you are running unix) or something
similar that would use awk or what else to read the file and update the
database during some night time hours when everyone is sleeping and not
watching? ;-)

konstantin

"Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
news:78pvd.2867$2J2.666@newsread2.news.atl.earthli nk.net...[color=blue]
>
> Let me explain, there is a zipped text file of about 72000 lines (with zip
> code data), that must be added to the DB. Once it gets to the server, it
> is unzipped, and the zip codes are updated and/or added.
>
> -d[/color]

re: script execution time too long

Zip codes are updated and changed all the time.

So, PHP scripts run by cron have no time limit?

-d

konsu wrote:
[color=blue]
> i see ;-)
>
> would you explain why you are processing the file as part of a web request?
> (i am still assuming you have a web request ;-)) is there a user who might
> want to submit such a file from a web page?
>
> can you create a cron job (assuming you are running unix) or something
> similar that would use awk or what else to read the file and update the
> database during some night time hours when everyone is sleeping and not
> watching? ;-)
>
> konstantin
>
> "Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
> news:78pvd.2867$2J2.666@newsread2.news.atl.earthli nk.net...
>[color=green]
>>Let me explain, there is a zipped text file of about 72000 lines (with zip
>>code data), that must be added to the DB. Once it gets to the server, it
>>is unzipped, and the zip codes are updated and/or added.
>>
>>-d[/color]
>
>
>[/color]

re: script execution time too long

> So, PHP scripts run by cron have no time limit?

i do not know about that but i think you have a better control over it. and
if you do not you can always use a non-PHP solution.

re: script execution time too long

"Doug" <dougd99@XXXXremovetheXearthlink.net> wrote in message
news:Aw9vd.1969$Yj4.1819@newsread3.news.atl.earthl ink.net...[color=blue]
>
>
> I have a script that does a huge database update, which takes a long
> time. However, the host I am running on (and I suspect most hosts) are
> running in "safe mode." This means I cannot change the maximum execution
> time of scripts. So, the script stops right in the middle of doing the
> update.
>
> On top of that, I am trying to make code that I can distribute to
> others. As of now, the only way that this script can be run is with sys
> admin intervention. I would like to make it run without any help from
> the system administrator.
>
> Does any one have a good solution for this problem?
>
> -d[/color]

Well, break up the process into multiple pieces. The initial request would
process the first X number of lines, then tell the client to refresh
(Javascript). The second request then process the next X number of lines and
so on.

It's a good way to indicate to the user that something is happening too.