By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
444,199 Members | 1,064 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 444,199 IT Pros & Developers. It's quick & easy.

efficient way for building class browsers.

P: n/a
I am looking for a class browser that has these features.

1. Given a symbol (class, method or function) it should giveback
the lineno n source code
2. It should be efficient and quick.

I looked at the `pyclbr` module, but it's slower. I will be _often_
regenerating the symbols from exactly one python source file. But for
an average python script in my machine it takes 2 secs for pyclbr to
build the dictionary of symbols.

When I profiled pyclbr, I found that most of the time was spent on the
tokenize.get_token** functions, and the remaining time was spent on
pyclbr._readmodule function.

Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )

Are there any better ways for achieving this?
Jul 18 '05 #1
Share this Question
Share on Google+
6 Replies


P: n/a
> I am looking for a class browser that has these features.

Only a browser? Text or GUI? What OS? Any particular GUI toolkit?
Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )
compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.
Are there any better ways for achieving this?


You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.

It wouldn't be too bad to pull the tree portion out, what kind of
browser do you need?

- Josiah
Jul 18 '05 #2

P: n/a
Josiah Carlson <jc******@nospam.uci.edu> wrote in message news:<c0**********@news.service.uci.edu>...
I am looking for a class browser that has these features.


Only a browser? Text or GUI? What OS? Any particular GUI toolkit?
Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )


compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.
Are there any better ways for achieving this?


You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.


Thats slow i have a parser that parses 100k in less then 15 ms on a
1,2 GHz system. It will be part of "http://www.python-ide.com"
It is a 100% GNU Eiffel solution which generates plain C code.

I think the 1/2 second could also be done with some intelligent
regexpr's. There is something like this in the IDLE code.
Jul 18 '05 #3

P: n/a
>>You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.
Thats slow i have a parser that parses 100k in less then 15 ms on a
1,2 GHz system. It will be part of "http://www.python-ide.com"
It is a 100% GNU Eiffel solution which generates plain C code.


I never said it was the fastest, I said it was "pretty fast", and
provided results that it was faster than the pyclbr and compiler.ast
modules.

It is also written in Python, which seems to be more of what Sridhar was
looking for. Furthermore, the source for the fast parser is freely
available. Considering that Arachno Python IDE is a commercial effort,
the liklihood of the parser being available in source or loadable module
format seems fairly unlikely.

I think the 1/2 second could also be done with some intelligent
regexpr's. There is something like this in the IDLE code.


Checking the IDLE source for version 1.0 (I would check CVS, but CVS
seems to be having issues at sourceforge), it uses pyclbr.

- Josiah

Jul 18 '05 #4

P: n/a
Josiah Carlson <jc******@nospam.uci.edu> wrote in message news:<c0**********@news.service.uci.edu>...
I think the 1/2 second could also be done with some intelligent
regexpr's. There is something like this in the IDLE code.


Checking the IDLE source for version 1.0 (I would check CVS, but CVS
seems to be having issues at sourceforge), it uses pyclbr.


The inspect module uses regexps to extract the source code from classes
(and it is always a very good reading).

Michele
Jul 18 '05 #5

P: n/a
Josiah Carlson <jc******@nospam.uci.edu> wrote in message news:<c0**********@news.service.uci.edu>...
I am looking for a class browser that has these features.
Only a browser? Text or GUI? What OS? Any particular GUI toolkit?


IDE. Linux,Windows,Mac. PyGTK
Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )


compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.
Are there any better ways for achieving this?


You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.


I'll try that. I have come across tagmanager (C) code written for
the Anjuta (http://anjuta.org) project. I should give it a try too.
It wouldn't be too bad to pull the tree portion out, what kind of
browser do you need?
I will put it in brief. The class browser can't support all
standard and third party python packages, since most of them is
available as shared objects. So mine is focused only on the current
project, i.e. user created (or application) python files. So the
browser should be able to fetch the tags (class, functions, methods)
from source code along with the line number information. That's all
about. Any further possiblities in this regard is also expected.
- Josiah

Jul 18 '05 #6

P: n/a
>>>I am looking for a class browser that has these features.

Only a browser? Text or GUI? What OS? Any particular GUI toolkit?
IDE. Linux,Windows,Mac. PyGTK


Seemingly this suggests that you want an IDE that runs on linux,
windows, mac, and that uses the PyGTK toolkit.

I personally don't spend much time looking through PyGTK apps, so I
can't be of much help. On the other hand, if you have a parser,
constructing the tree control/widget from the parsing is very easy.

You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourceforge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.


I'll try that. I have come across tagmanager (C) code written for
the Anjuta (http://anjuta.org) project. I should give it a try too.


The control in PyPE is written for wxPython, but the parser is platform
independant.

I will put it in brief. The class browser can't support all
standard and third party python packages, since most of them is
available as shared objects. So mine is focused only on the current
project, i.e. user created (or application) python files. So the
browser should be able to fetch the tags (class, functions, methods)
from source code along with the line number information. That's all
about. Any further possiblities in this regard is also expected.


Fetching the various function/class/method names is easy, look at the
parser included with PyPE. It isn't perfect, but works pretty well
considering that it is neither a regular expression, nor does it have a
tokenizer.

- Josiah
Jul 18 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.