Tracing the execution of scripts?

[Michael B. Trausch]

Alright, I seem to be at a loss for what I am looking for, and I am not
even really all that sure if it is possible or not. I found the 'pdb'
debugger, but I was wondering if there was something that would trace or
log the order of line execution for a multi-module Python program. I am
having a little bit of a problem tracking down a problem that I
mentioned earlier
(http://groups.google.com/group/comp....59fc888b365be),
and I am hoping that maybe seeing how the statements are executed will
shed some light on the entire thing for me.

The debugger isn't working, though, because I have a timer set up to
fire off every 20ms to check for incoming network traffic from the
server, and that timer firing off makes stepping through with the
debugger seemingly impossible to get any useful data.

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting? I know that the
server is sending and receiving properly, because I can connect to it
with a regular telnet client and it works -- but at the same time,
everything I can think of to check my program's functionality checks out
just fine (e.g., it reports that it is successfully sending what I am
typing to the server, and it says that the server is not sending
anything back to be read from the socket).

If it makes any difference, I am using wxWidgets' Python bindings for
the UI code (and the timer), and using Python's socket library for the
network stuff.

-- Mike

[Ben Finney]

Jean-Paul Calderone <ex*****@divmod .comwrites:

1) Write unit tests for your code. Keep writing unit tests until
you have some that _don't pass_. Then fix your code so that they
do. When you do further development, write the tests first, then
implement the code that makes them pass.

Hear hear.

Be advised, though, that attempting to apply unit tests to code that
wasn't designed with testing in mind may very quickly reveal a poor
design. [0] If you can't easily test pieces of the code independently,
you probably haven't written those pieces to be loosely coupled and
well-defined.

The moral? Writing unit tests *along with* the functional code will
result in a design that is loosely coupled, and probably
better-defined. Also, hopefully, easy to test :-)
[0] I have no idea whether this is the case for the OP. It's a very
common symptom that arises from people who are first advised to
introduce tests to their code, though.

--
\ "If you ever catch on fire, try to avoid seeing yourself in the |
`\ mirror, because I bet that's what REALLY throws you into a |
_o__) panic." -- Jack Handey |
Ben Finney

[Stephan Kuhagen]

"Michael B. Trausch" <"mike$#at^&nos pam!%trauschus" wrote:

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting?

Python itself can do this for you. A __VERY__ simple approach:

|def myCallTrace (frame, event, arg):
| tracefile.write ('localTrace\n frame: '+str(frame)+'\ n event:
|'+str(event)+' \n arg: '+str(arg)+'\n' )
| tracefile.write (' '+str(inspect.g etframeinfo(fra me))+'\n')
|
|def myTrace (frame, event, arg):
| tracefile.write ('globalTrace\n frame: '+str(frame)+'\ n event:
|'+str(event)+' \n arg: '+str(arg)+'\n' )
| if event == 'call':
| return myCallTrace
|
|import inspect
|import sys
|
|sys.settrace(m yTrace)
|tracefile = file('trace.txt ', 'w')

Insert this in you program and you get a trace of every line of Python-Code
executed in the file trace.txt. You must read the documentation of the
module inspect and of sys.settrace() to understand, what happens and what
it means. Additionally, if it should work with threads, you must take care
that every thread gets its own output file. But as a first step to get a
trace of execution, this should do it.

HTH, Regards
Stephan

[R. Bernstein]

pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines
as they are executed as well as an --output option to have this sent
to a file rather than stdout. If your program has threads it would be
good to use the --threading option. (The best threading support is if
your program uses the threading module for threads rather than the
lower-level thread module.)

A demo of the debugger including showing line tracing and output to a file is
here:
http://showmedo.com/videos/video?nam...romSeriesID=28
"Michael B. Trausch" <"mike$#at^&nos pam!%trauschus" writes:

Alright, I seem to be at a loss for what I am looking for, and I am not
even really all that sure if it is possible or not. I found the 'pdb'
debugger, but I was wondering if there was something that would trace or
log the order of line execution for a multi-module Python program. I am
having a little bit of a problem tracking down a problem that I
mentioned earlier
(http://groups.google.com/group/comp....59fc888b365be),
and I am hoping that maybe seeing how the statements are executed will
shed some light on the entire thing for me.

The debugger isn't working, though, because I have a timer set up to
fire off every 20ms to check for incoming network traffic from the
server, and that timer firing off makes stepping through with the
debugger seemingly impossible to get any useful data.

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting? I know that the
server is sending and receiving properly, because I can connect to it
with a regular telnet client and it works -- but at the same time,
everything I can think of to check my program's functionality checks out
just fine (e.g., it reports that it is successfully sending what I am
typing to the server, and it says that the server is not sending
anything back to be read from the socket).

If it makes any difference, I am using wxWidgets' Python bindings for
the UI code (and the timer), and using Python's socket library for the
network stuff.

-- Mike

[Michael B. Trausch]

Ben Finney wrote:

Jean-Paul Calderone <ex*****@divmod .comwrites:

>1) Write unit tests for your code. Keep writing unit tests until
you have some that _don't pass_. Then fix your code so that they
do. When you do further development, write the tests first, then
implement the code that makes them pass.

Hear hear.

Be advised, though, that attempting to apply unit tests to code that
wasn't designed with testing in mind may very quickly reveal a poor
design. [0] If you can't easily test pieces of the code independently,
you probably haven't written those pieces to be loosely coupled and
well-defined.

I will whole-heartedly admit that my code is probably poorly designed.
*shrug* Though, I *do* try to write everything in such a way as to be
able to easily re-use it later. When I wrote little things just to help
me sysadmin, I learned that was really key to making my life easier.

Besides, I am lazy... reuse certainly serves my needs there! :-)

>
The moral? Writing unit tests *along with* the functional code will
result in a design that is loosely coupled, and probably
better-defined. Also, hopefully, easy to test :-)

I think I have more to learn on the concept of unit-testing. I have
never seen a group push the idea so hard. I have read about unit
testing before, and even written tests (in other languages) to test
modules of library code that I put together once, but that was about it.
I need to, I think, get more "into" the concept, though. It isn't
something that I am able to just whip out and go "Hey, a widget unit
test! And it works!" probably because of my own lack of practice.

>
[0] I have no idea whether this is the case for the OP. It's a very
common symptom that arises from people who are first advised to
introduce tests to their code, though.

It is very likely that it is the case. I am, by all means, what I could
consider to be a novice programmer when it comes to anything outside of
my little world of PHP. I can write web applications in PHP until the
cows come home with little to no problem, and I can play with SQL the
same way. But, I seem to have problems when I step out of that little
box, and so, that tells me that I need to work harder at it. :-)

-- Mike

[Michael B. Trausch]

Stephan Kuhagen wrote:

"Michael B. Trausch" <"mike$#at^&nos pam!%trauschus" wrote:

>Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting?

Python itself can do this for you. A __VERY__ simple approach:

[snip]

>
Insert this in you program and you get a trace of every line of Python-Code
executed in the file trace.txt. You must read the documentation of the
module inspect and of sys.settrace() to understand, what happens and what
it means. Additionally, if it should work with threads, you must take care
that every thread gets its own output file. But as a first step to get a
trace of execution, this should do it.

I will need to keep this around to look at a little later. It looks
like it would be something particularly useful when line-tracing by
itself fails. I am constantly finding myself amazed at Python's
capabilities.

-- Mike

[Fulvio]

*************** ********
Your mail has been scanned by InterScan MSS.
*************** ********
On Friday 27 October 2006 17:31, R. Bernstein wrote:

pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines

I faced several time that pydb stuck without sign of errors. In the other hand
Pdb doesn't appear that problem.
Mostly pydb freeze on long loops.
It might be some problem on my setup, I'll check it up...

F

[Larry Bates]

I just thought I would put my 2 cents in on this issue. Others
have suggested that unit tests are an excellent way of debugging
your code and I agree. I also find that writing code from the
outset using a logging class (there is one in the standard
library) that allows you to create log files of information as
you run your application is a good idea. Using increasingly
detailed logs (I use a debug mode of 1,2,3,4 for more detail)
that dump where you are and intermediate variable values works
EXTREMELY well for me. I leave this code in the application
so that I can have customers (or myself) run the application
in debug mode should I have a hard to find problem. This is
especially true for long running or lights-out batch apps
that have no UI making debugging even more difficult. I find
that the overhead of testing if I'm in debug mode and logging
results is almost non-existent to the overall execution speed
of my scripts, but then I don't have very many really speed
sensitive scripts so your mileage might vary.

Hope the information helps.

-Larry

[fumanchu]

Stephan Kuhagen wrote:

"Michael B. Trausch" <"mike$#at^&nos pam!%trauschus" wrote:

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting?

Python itself can do this for you. A __VERY__ simple approach:
...
Additionally, if it should work with threads, you must take care
that every thread gets its own output file.

Or be differentiated *somehow*. My pyconquer module tabs each thread's
activity into swimlanes, so you can even see when each thread starts,
obtains/releases the GIL, and stops. See the longer example here:
http://projects.amor.org/misc/wiki/PyConquer
Robert Brewer
System Architect
Amor Ministries
fu******@amor.o rg

[R. Bernstein]

Fulvio <fu****@tm.net. mywrites:

*************** ********
Your mail has been scanned by InterScan MSS.
*************** ********

Delighted to know that.

>
On Friday 27 October 2006 17:31, R. Bernstein wrote:

pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines

I faced several time that pydb stuck without sign of errors. In the other hand
Pdb doesn't appear that problem.
Mostly pydb freeze on long loops.

In version 1.19 (released today), I extended signal handling so that
you can send the program a signal and it will print a stack trace of
where it is and continue. But even before this release, you could
arrange to enter the debugger by sending it a signal. It's possible
something like this might be used help track down the problem in
either pydb or another Python program that seems to be not
responding. On the other hand, use at your own risk - I don't
guarentee this will work for you.

And.. before you ask for more details, I'll repeat what someone else
posted in response to another of your questions:

I guess you've been told to read this here, but just in case it
wasn't, or you didn't bother to read it:

http://catb.org/esr/faqs/smart-questions.html

It might be some problem on my setup, I'll check it up...

Given your other posts, that's quite possible. If it's not, submit a
bug report. (Posting to c.l.r isn't the same as submitting a bug
report). Thanks.