Resuming a program's execution after correcting error

[Sheldon]

Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

Sincerely,
Sheldon

[Georg Brandl]

Sheldon wrote:

Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

As I said before, this can be done by finding out where the error is raised,
what the cause is and by inserting an appropriate try-except-statement in
the code.

There's no way to do that without modifying the code, if the program itself
doesn't offer such an option.

Georg

[Tuomas]

Sheldon wrote:

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

May be you could put some parameter to your main loop:

start_point=begin
if start_point==begin: output=open('outputfile', 'w')
else : output=open('outputfile', 'a')

while start_point <= case <= end_point:

try:
do_complex_computation(case)
except Exception:
print case
break

If you get an error you repair the program and set

start_point=case

and go on with the program.

Tuomas

[Dan Bishop]

Sheldon wrote:

Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

Afaik, the best you can do is run the script with "python -i", which
will give you an interactive prompt from which you have access to the
module-level variables. Or even better, test your script with small
data sets, and do the real calculations only after you've fixed any
obvious bugs.

[Jay]

I don't know how much help this is going to be, but you could store
values in a temporary file on the hard disk and write checkpoints to
read in the data and begin from a point somewhere in the middle of the
script.

Sheldon wrote:

Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

Sincerely,
Sheldon

[MonkeeSage]

Georg Brandl wrote:

As I said before, this can be done by finding out where the error is raised,
what the cause is and by inserting an appropriate try-except-statement in
the code.

I could be mistaken, but I *think* the OP is asking how to re-enter the
stack at the same point as the exception exited from and continue with
the execution as if the exception never happened. AFAIK, that isn't
possible; however, given that he has a file to work from that indicates
a portion of the state at the time of the exception, I think he may be
able simulate that kind of functionality by reading in the file on
exception and then returning a call to the function where the exception
occured with the data from the file. Something like this mockup:

def faulty_function(a, b, c=None):
if not c:
c = 0
try:
# modify c, write c to file...
# oops hit an exception
c += a / b
except:
# read from the file here
# c = ...
# and fix the error
b += 1
return faulty_function(a, b, c)
return c

print faulty_function(2, 0) # =2

Of course, it's probably much better to just fix the code and avoid the
exception in the first place. ;)

Regards,
Jordan

[MRAB]

Sheldon wrote:

Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

You could modify the program while you're debugging it so that instead
of, say:

calculate data
write data

you have:

if saved data exists:
load data
else:
calculate data
save data
write data

The pickle module would be useful here.

Matthew

[Sheldon]

MonkeeSage wrote:

Georg Brandl wrote:

As I said before, this can be done by finding out where the error is raised,
what the cause is and by inserting an appropriate try-except-statement in
the code.

I could be mistaken, but I *think* the OP is asking how to re-enter the
stack at the same point as the exception exited from and continue with
the execution as if the exception never happened. AFAIK, that isn't
possible; however, given that he has a file to work from that indicates
a portion of the state at the time of the exception, I think he may be
able simulate that kind of functionality by reading in the file on
exception and then returning a call to the function where the exception
occured with the data from the file. Something like this mockup:

def faulty_function(a, b, c=None):
if not c:
c = 0
try:
# modify c, write c to file...
# oops hit an exception
c += a / b
except:
# read from the file here
# c = ...
# and fix the error
b += 1
return faulty_function(a, b, c)
return c

print faulty_function(2, 0) # =2

Of course, it's probably much better to just fix the code and avoid the
exception in the first place. ;)

Regards,
Jordan

Thanks Jordon,
I think you understood my problem best. I know now that this is not
possible but I would like to create an exception that saves all the
current variables when there is an error. I think pickle is the answer
but I never got it to work. My program is very large and it is being
modified often.
Any advice on how to save the variables.

/Sheldon

[Sheldon]

MRAB wrote:

Sheldon wrote:

Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

You could modify the program while you're debugging it so that instead
of, say:

calculate data
write data

you have:

if saved data exists:
load data
else:
calculate data
save data
write data

The pickle module would be useful here.

Matthew

I like your idea Matthew but I don't know how to pickle the many
variables in one file. Do I need to pickle each and every variable into
a seperate file?
var1,var2
pickle.dump(var1,f)
pickle.dump(var2,f2)

/Sheldon

[MRAB]

Sheldon wrote:

MRAB wrote:

Sheldon wrote:

Hi.
>
Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.
>

You could modify the program while you're debugging it so that instead
of, say:

calculate data
write data

you have:

if saved data exists:
load data
else:
calculate data
save data
write data

The pickle module would be useful here.

Matthew

I like your idea Matthew but I don't know how to pickle the many
variables in one file. Do I need to pickle each and every variable into
a seperate file?
var1,var2
pickle.dump(var1,f)
pickle.dump(var2,f2)

Using the 'pickle' module:

# To store:
f = open(file_path, "wb")
pickle.dump(var1, f)
pickle.dump(var2, f)
f.close()

# To load
f = open(file_path, "rb")
var1 = pickle.load(f)
var2 = pickle.load(f)
f.close()

A more flexible alternative is to use the 'shelve' module. This behaves
like a dict:

# To store
s = shelve.open(file_path)
s["var1"] = "first"
s["var2"] = [2, 3]
s.close()

# To load
s = shelve.open(file_path)
print s["var1"] # This prints "first"
print s["var2"] # This prints [2, 3]
s.close()

Hope that helps
Matthew

[hanumizzle]

On 3 Oct 2006 16:58:17 -0700, MRAB <go****@mrabarnett.plus.comwrote:

I like your idea Matthew but I don't know how to pickle the many
variables in one file. Do I need to pickle each and every variable into
a seperate file?
var1,var2
pickle.dump(var1,f)
pickle.dump(var2,f2)

Using the 'pickle' module:

# To store:
f = open(file_path, "wb")
pickle.dump(var1, f)
pickle.dump(var2, f)
f.close()

# To load
f = open(file_path, "rb")
var1 = pickle.load(f)
var2 = pickle.load(f)
f.close()

A more flexible alternative is to use the 'shelve' module. This behaves
like a dict:

# To store
s = shelve.open(file_path)
s["var1"] = "first"
s["var2"] = [2, 3]
s.close()

# To load
s = shelve.open(file_path)
print s["var1"] # This prints "first"
print s["var2"] # This prints [2, 3]
s.close()

As long as we're on the subject of data serialization, I should like
to bring up PyYaml. YAML is a portable format for storing data of all
kinds; it became popular via Ruby I think, but there are
implementations for many other languages. If you stick to storing
simple stuff like lists, strings, and dictionaries, you can use your
YAML data almost anywhere, but PyYaml even supports reifying things
like lambdas.

[Sheldon]

MRAB wrote:

Sheldon wrote:

MRAB wrote:

Sheldon wrote:
Hi.

Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.

You could modify the program while you're debugging it so that instead
of, say:
>
calculate data
write data
>
you have:
>
if saved data exists:
load data
else:
calculate data
save data
write data
>
The pickle module would be useful here.
>
Matthew

I like your idea Matthew but I don't know how to pickle the many
variables in one file. Do I need to pickle each and every variable into
a seperate file?
var1,var2
pickle.dump(var1,f)
pickle.dump(var2,f2)

Using the 'pickle' module:

# To store:
f = open(file_path, "wb")
pickle.dump(var1, f)
pickle.dump(var2, f)
f.close()

# To load
f = open(file_path, "rb")
var1 = pickle.load(f)
var2 = pickle.load(f)
f.close()

A more flexible alternative is to use the 'shelve' module. This behaves
like a dict:

# To store
s = shelve.open(file_path)
s["var1"] = "first"
s["var2"] = [2, 3]
s.close()

# To load
s = shelve.open(file_path)
print s["var1"] # This prints "first"
print s["var2"] # This prints [2, 3]
s.close()

Hope that helps
Matthew

Perfect Matthew!

Much obliged!

/Sheldon

[Henning Hasemann]

Sheldon wrote:

MRAB wrote:

>Sheldon wrote:

>>MRAB wrote:
Sheldon wrote:
Hi.
>
Does anyone know if one can resume a python script at the error point
after the error is corrected?
I have a large program that take forever if I have to restart from
scratch everytime. The error was the data writing a file so it seemed
such a waste if all the data was lost and must be recalculated again.
>

Sorry if this is off-topic here but Dylan (http://www.opendylan.org) is
a nice language (I sometimes like even more than python itself) that
allows you to continue the work right where the exception was thrown.

Henning

[Scott David Daniels]

Henning Hasemann wrote:

Sheldon wrote:

....

>>>>>Does anyone know if one can resume a python script at the error point
>after the error is corrected?
>I have a large program that take forever if I have to restart from
>scratch everytime. The error was the data writing a file so it seemed
>such a waste if all the data was lost and must be recalculated again.

Sorry if this is off-topic here but Dylan (http://www.opendylan.org) is
a nice language (I sometimes like even more than python itself) that
allows you to continue the work right where the exception was thrown.

As I have explained before in this newsgroup, Xerox Parc had that
ability (the ability to finish an exception by returning to its source)
in their system implementation language, and finally removed the
capability when they saw how many bugs were related to its use.

--Scott David Daniels
sc***********@acm.org