469,352 Members | 1,661 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,352 developers. It's quick & easy.

Python AST preserving whitespace and comments

Hi,

I'm working on Pythoscope[1], a unit test generator for Python and
stumbled into the following problem. I need a way to analyze and
modify Python AST tree, but without loosing source code formatting and
comments. Standard library ast module discards those, so I started
looking for other solutions. I found a library used by the 2to3
script[2], which does different kinds of source code refactorings and
preserves the formatting of the original file. Sadly AST generated by
this library is not compatible with the standard one, so I can't use a
nice interface of compiler.visitor[3]. It isn't also as well
documented. I kind of liked the clean and descriptive list of standard
AST nodes[4].

So here are my questions to the list. Is there any library built on
top of lib2to3 which makes traversing it easier? The pattern matcher
is nice, but not always feasible for the higher level view of the
source code.

Are there any tutorials/documentation about usage of lib2to3 which I'm
not aware of? I basically read through HACKING, README and a lot of
source code, but would appreciate some more examples and insights.

Is the lib2to3 library meant to be used outside of the 2to3 script or
rather not? Are there any projects that will incorporate its features
(enriched AST in particular) in a more "official" library-like
package? Will lib2to3 be put on PyPI anytime?

[1] http://pythoscope.org/
[2] http://svn.python.org/view/sandbox/trunk/2to3/
[3] http://docs.python.org/lib/module-compiler.visitor.html
[4] http://docs.python.org/lib/module-compiler.ast.html

Cheers,
mk
Aug 28 '08 #1
1 1866


Michal Kwiatkowski wrote:
Hi,

I'm working on Pythoscope[1], a unit test generator for Python and
stumbled into the following problem. I need a way to analyze and
modify Python AST tree, but without loosing source code formatting and
comments. Standard library ast module discards those, so I started
looking for other solutions. I found a library used by the 2to3
script[2], which does different kinds of source code refactorings and
preserves the formatting of the original file. Sadly AST generated by
this library is not compatible with the standard one, so I can't use a
nice interface of compiler.visitor[3]. It isn't also as well
documented. I kind of liked the clean and descriptive list of standard
AST nodes[4].

So here are my questions to the list. Is there any library built on
top of lib2to3 which makes traversing it easier? The pattern matcher
is nice, but not always feasible for the higher level view of the
source code.

Are there any tutorials/documentation about usage of lib2to3 which I'm
not aware of? I basically read through HACKING, README and a lot of
source code, but would appreciate some more examples and insights.

Is the lib2to3 library meant to be used outside of the 2to3 script or
rather not? Are there any projects that will incorporate its features
(enriched AST in particular) in a more "official" library-like
package? Will lib2to3 be put on PyPI anytime?

[1] http://pythoscope.org/
[2] http://svn.python.org/view/sandbox/trunk/2to3/
[3] http://docs.python.org/lib/module-compiler.visitor.html
[4] http://docs.python.org/lib/module-compiler.ast.html
You found the appropriate library, but I am unaware of anything that you
missed. I think others have had the idea that 2to3 might grow into
something more, but the core developers are currently focues on getting
2.6/3.0 out by the end of September.

Aug 28 '08 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

699 posts views Thread by mike420 | last post: by
34 posts views Thread by Brandon J. Van Every | last post: by
8 posts views Thread by jerry.levan | last post: by
9 posts views Thread by amattie | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.