C parsing fun

[karoly.kiripolszky]

Helo ppl!

At the job I was given the task to make a script to analyze C++ code
based on concepts my boss had. To do this I needed to represent C++
code structure in Python somehow. I read the docs for Yapps, pyparsing
and other stuff like those, then I came up with a very simple idea. I
realized that bracketed code is almost like a Python list, except I
have to replace curly brackets with squared ones and surround the
remaining stuff with quotes. This process invokes no recursion or node
objects, only pure string manipulations so I believe it's really fast.
Finally I can get the resulting list by calling eval() with the
string.

For example when I need to parse a class definition, I only need to
look for a list item containing the pattern "*class*", and the next
item will be the contents of the class as another list.

You can grab the code at:

http://kiri.csing.hu/stack/python/bloppy-0.1.zip

(test script [test.py] included)

[karoly.kiripolszky]

and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.

[Szabolcs Nagy]

based on concepts my boss had. To do this I needed to represent C++

code structure in Python somehow. I read the docs for Yapps, pyparsing
and other stuff like those, then I came up with a very simple idea. I
realized that bracketed code is almost like a Python list, except I
have to replace curly brackets with squared ones and surround the
remaining stuff with quotes. This process invokes no recursion or node

yes that's a nice solution
sometimes it's not enough though (won't work on code obfuscated with
macros)

anyway if you need something more sophisticated then i'd recommend
gccxml or it's python binding:

http://www.language-binding.net/pygccxml/pygccxml.html

[=?iso-8859-1?q?K=E1roly_Kiripolszky?=]

Thx for responding, Szabolcs! I've already tried that, but couldn't
manage to get it to work. The source I tried to parse is a huge MSVC
7.1 solution containing about 38 projects, and I believe the code is
so complex that it has too many different dependencies and GCC just
can't handle them. Btw I'm not deeply familiar with C++ compilers, so
maybe it was because of compiler misconfiguratio n, but I really don't
know...

Szabolcs Nagy írta:

based on concepts my boss had. To do this I needed to represent C++
code structure in Python somehow. I read the docs for Yapps, pyparsing
and other stuff like those, then I came up with a very simple idea. I
realized that bracketed code is almost like a Python list, except I
have to replace curly brackets with squared ones and surround the
remaining stuff with quotes. This process invokes no recursion or node

yes that's a nice solution
sometimes it's not enough though (won't work on code obfuscated with
macros)

anyway if you need something more sophisticated then i'd recommend
gccxml or it's python binding:

http://www.language-binding.net/pygccxml/pygccxml.html

[Marc 'BlackJack' Rintsch]

In <11************ **********@v33g 2000cwv.googleg roups.com>,
karoly.kiripols zky wrote:

and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.

But it fails if brackets appear in comments or literal strings.

Ciao,
Marc 'BlackJack' Rintsch

[=?iso-8859-1?q?K=E1roly_Kiripolszky?=]

Marc 'BlackJack' Rintsch írta:

In <11************ **********@v33g 2000cwv.googleg roups.com>,
karoly.kiripols zky wrote:

and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.

But it fails if brackets appear in comments or literal strings.

Ciao,
Marc 'BlackJack' Rintsch

[=?iso-8859-1?q?K=E1roly_Kiripolszky?=]

You're right, thank you for the comment! I will look after how to
avoid this.

Marc 'BlackJack' Rintsch írta:

In <11************ **********@v33g 2000cwv.googleg roups.com>,
karoly.kiripols zky wrote:

and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.

But it fails if brackets appear in comments or literal strings.

Ciao,
Marc 'BlackJack' Rintsch

[Claudio Grondi]

Károly Kiripolszky wrote:

You're right, thank you for the comment! I will look after how to
avoid this.

And after you have resolved this 'small' ;-) detail you will probably
notice, that some full functional and in wide use being parser have
still trouble with this ...

Claudio

>
Marc 'BlackJack' Rintsch írta:

>In <11************ **********@v33g 2000cwv.googleg roups.com>,
karoly.kiripol szky wrote:

>>and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.

But it fails if brackets appear in comments or literal strings.

Ciao,
Marc 'BlackJack' Rintsch

[=?iso-8859-1?q?K=E1roly_Kiripolszky?=]

I've found a brute-force solution. In the preprocessing phase I simply
strip out the comments (things inside comments won't appear in the
result) and replace curly brackets with these symbols: #::OPEN::# and
#::CLOSE::#. After parsing I convert them back. In fact I can disclude
commented lines from the analyzis as I only have to cope with
production code.

Claudio Grondi írta:

Károly Kiripolszky wrote:

You're right, thank you for the comment! I will look after how to
avoid this.

And after you have resolved this 'small' ;-) detail you will probably
notice, that some full functional and in wide use being parser have
still trouble with this ...

Claudio

Marc 'BlackJack' Rintsch írta:

In <11************ **********@v33g 2000cwv.googleg roups.com>,
karoly.kiripols zky wrote:

and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.
But it fails if brackets appear in comments or literal strings.

Ciao,
Marc 'BlackJack' Rintsch

[=?iso-8859-1?q?K=E1roly_Kiripolszky?=]

http://kiri.csing.hu/stack/python/bloppy-0.2.zip

Test data now also contains brackets in literal strings.

Claudio Grondi írta:

Károly Kiripolszky wrote:

You're right, thank you for the comment! I will look after how to
avoid this.

And after you have resolved this 'small' ;-) detail you will probably
notice, that some full functional and in wide use being parser have
still trouble with this ...

Claudio

Marc 'BlackJack' Rintsch írta:

In <11************ **********@v33g 2000cwv.googleg roups.com>,
karoly.kiripols zky wrote:

and the great thing is that the algorithm can be used with any
language that structures the code with brackets, like PHP and many
others.
But it fails if brackets appear in comments or literal strings.

Ciao,
Marc 'BlackJack' Rintsch