Long execution time for a script: avoid error messages ?

[Damien]

Hello to all,
I've written a script that take quite a long time to execute (email
sending). Last time I ran it, PHP gave me a "execution time limit
exceeded" (or something like that), so I changed the value in php.ini.

Now, the questions are :
- How can I make sure I'm not going to have problems when execution time
lengthens still ? I'm afraid to set too high a time limit in case other
scripts run astray.
- Can I "break" the job into "batches" ?

I'm going to try to see what takes som much time, but I'm afraid it is
always going to be long.

Anyway, thanks in advance for any help !

BR,
Damien

[Gert]

> Now, the questions are :

- How can I make sure I'm not going to have problems when execution time
lengthens still ? I'm afraid to set too high a time limit in case other
scripts run astray.
I think that depends on what your script does... in case you've got a loop
somewhere
you could save a timestamp when the script starts (before the loop) and
within you
compare the actual timestamp with the first one... of course this also costs
time... I've never
tried that but I think it could work

- Can I "break" the job into "batches" ?

Neither tried this, but what about - if your time-stamp-comparison tells you
that the script is about
to stop due to a timeout, you send a header(Location.
path_to_script.php?id=5) to let your
script know that it should continue with id number 5
Has anyone experiences with such a strategy??

Greetings,

Gert.

[Ian.H]

On Mon, 13 Sep 2004 11:56:25 +0200, Damien <tr**@bidule.net> wrote:

Hello to all,
I've written a script that take quite a long time to execute (email
sending). Last time I ran it, PHP gave me a "execution time limit
exceeded" (or something like that), so I changed the value in php.ini.

Now, the questions are :
- How can I make sure I'm not going to have problems when execution time
lengthens still ? I'm afraid to set too high a time limit in case other
scripts run astray.
- Can I "break" the job into "batches" ?

I'm going to try to see what takes som much time, but I'm afraid it is
always going to be long.

Anyway, thanks in advance for any help !

BR,
Damien

You could break it into batches.. I did this a week ago for a client but
may depend on exactly what kind of queries are being run. If they're
relatively "simple"[1] there should be no problem doing that.

I wrote a simple "pagination" function that then worked out the starting
record for the SQL query and displayed a 'prev | next' page link (I
actually used the code from a Smarty plugin I wrote.. the 'prev' link
may not be necessary etc) but this then worked no problem for about
15,000 entries =)

Regards,

Ian
[1]: Referring to "simple" here rather than one query I had that didn't
work so well like this as it was a recursive loop to query too but if
it's "straight forward" (maybe a better description than "simple") there
should be no real issues.

--
Ian.H
digiServ Network
London, UK
http://digiserv.net/

[Damien]

Ian.H a écrit :
(snip)

You could break it into batches.. I did this a week ago for a client but
may depend on exactly what kind of queries are being run. If they're
relatively "simple"[1] there should be no problem doing that.
Actually, based on my "chronoscript", what's costing me time is the
mail() function, not the queries I make in order to prepare the mail
being sent.

I wrote a simple "pagination" function that then worked out the starting
record for the SQL query and displayed a 'prev | next' page link (I
actually used the code from a Smarty plugin I wrote.. the 'prev' link
may not be necessary etc) but this then worked no problem for about
15,000 entries =)

Problem with my script is that the loop generates the elements from what
is posted by a form: recipient list, message. So I'm afraid I'm going to
have troubles breaking the receipients into batches unless I store them
in a table first.

Hmmm, guess I've got some DB filling ahead ;) . Any other idea ?

Thanks for the tips anyway !
Damien
--
"And you told them that servers were like old people - you put them to
sleep at night and there's a few dead ones in the morning?"
BOFH

[Ian.H]

On Mon, 13 Sep 2004 17:55:36 +0200, Damien <tr**@bidule.net> wrote:

Ian.H a écrit :
(snip)

You could break it into batches.. I did this a week ago for a client but
may depend on exactly what kind of queries are being run. If they're
relatively "simple"[1] there should be no problem doing that.

Actually, based on my "chronoscript", what's costing me time is the
mail() function, not the queries I make in order to prepare the mail
being sent.

I wrote a simple "pagination" function that then worked out the starting
record for the SQL query and displayed a 'prev | next' page link (I
actually used the code from a Smarty plugin I wrote.. the 'prev' link
may not be necessary etc) but this then worked no problem for about
15,000 entries =)

Problem with my script is that the loop generates the elements from what
is posted by a form: recipient list, message. So I'm afraid I'm going to
have troubles breaking the receipients into batches unless I store them
in a table first.

Hmmm, guess I've got some DB filling ahead ;) . Any other idea ?

Thanks for the tips anyway !
Damien

Damien, are the bodies of the mail the same? or are they "personalised"?

I'm assuming that you're firing off multiple mails using mail() sending
by only the 'TO' header. Have you thought about the loop generating a CC
header? (or BCC.. but not sure how many these days reject BCC'd mail due
to abuse). This would result in just one mail actually being processed
to be sent but with multiple recipients which may cure your mail()
bottleneck.

There are a few issues to maybe take into account here though regarding
my suggestions.. as mentioned, BCC has become an "easy abuse method" and
some admins drop these mails. Likewise, too many recipients in the CC
header and the mail will also be rejected.

Not knowing the exact workings of your structure.. these ideas maybe
null&void.. but _might_ be an alternative option =)

Regards,

Ian

--
Ian.H
digiServ Network
London, UK
http://digiserv.net/

[Ian.H]

On Mon, 13 Sep 2004 16:24:09 GMT, Ian.H <ia*@WINDOZEdigiserv.net> wrote:
[ snip ]

There are a few issues to maybe take into account here though regarding
my suggestions.. as mentioned, BCC has become an "easy abuse method" and
some admins drop these mails. Likewise, too many recipients in the CC
header and the mail will also be rejected.

^^^^
[ snip ]
Oops! That should really be "may" rather than "will" =)

Regards,

Ian

--
Ian.H
digiServ Network
London, UK
http://digiserv.net/

[Damien]

Ian.H wrote:
(snip)

Damien, are the bodies of the mail the same? or are they "personalised"?

(snip)

Unfortunately, no... I'm trying to customize the message body
(recipient's name, trackers...) so I have to process it for every person.

Hmmm, I guess I'm going to design a temp table that holds the job ID and
the info for the recipients. Then I'll query this table in "batches" (eg
10 by 10 or so). Actually, this may make my life easier in several aspects.

Has anybody ever implemented something like that ? Anything I should be
aware of ? Like security, etc ? Not that anybody knows ANY PHP of SQL,
but... "A warned developer is worth 2, even when he's twins" :)

Thanks again for the input, your ideas on BCC-ing are pretty sensible,
I'm sure. As for CC-ing... I've just received the whole address book of
one of my suppliers informing us of a change of email :)

BR,
Damien
--

[Ian.H]

On Mon, 13 Sep 2004 19:19:35 +0200, Damien <tr**@bidule.net> wrote:

Ian.H wrote:
(snip)

Damien, are the bodies of the mail the same? or are they "personalised"?(snip)

Unfortunately, no... I'm trying to customize the message body
(recipient's name, trackers...) so I have to process it for every person.

Hmmm, I guess I'm going to design a temp table that holds the job ID and
the info for the recipients. Then I'll query this table in "batches" (eg
10 by 10 or so). Actually, this may make my life easier in several aspects.

Fair enough.. sometimes I've found a little more time spent on the
structure makes future development much easier and quicker.. hope all
goes well =)

Has anybody ever implemented something like that ? Anything I should be
aware of ? Like security, etc ? Not that anybody knows ANY PHP of SQL,
but... "A warned developer is worth 2, even when he's twins" :)

Can't say I personally have.. any "mailing list" style mails I use I run
through Mailman (but obviously this doesn't work for all situations).

Thanks again for the input, your ideas on BCC-ing are pretty sensible,
I'm sure. As for CC-ing... I've just received the whole address book of
one of my suppliers informing us of a change of email :)

BR,
Damien

Heh.. np =)

I actually drop mail here with a CC header of more than 10 addresses..
but I'm a draconian bastard when it comes to my mailservers (although I
do whitelist upon customer request if viable =) ) with only 1-2 spam
mails per couple of months received now (for all accounts, including
customers.. which are then squashed (well, tagged) by SpamAssassin and
then escalated to the DNSbl) rather than the 500+ to a single account I
used to get.

Regards,

Ian

--
Ian.H
digiServ Network
London, UK
http://digiserv.net/