473,320 Members | 1,799 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Deep Black Magic in Python: please help

Hi everyone,

I am trying to convince my managers that python can replace the outdated and
soon no-longer maintained proprietary system (Tool for Calculator Design) we
use here. In order to achieve this, I need to analyze python code which will
look somethink like this:

def foo(arg_dict):
return arg_dict["one"] + bar(arg_dict)

def bar(other_dict):
return other_dict["two"] + other_dict[True and "three" or "four"]

The result of my analysis should return a list

['one', 'two', 'three']

Alright, that doesn't sound so bad. With a little RTFM-ing and hacking, I got

-- black magic starts here --

import compiler

def _evaluate(partial, name=""):
from compiler.ast import Expression
from compiler.misc import set_filename
from compiler.pycodegen import ExpressionCodeGenerator

tree = Expression(partial)
set_filename(name, tree)
gen = ExpressionCodeGenerator(tree)
return eval(gen.getCode())

class RecursiveVisitor(compiler.visitor.ASTVisitor):
def __init__(self, name):
self.name = "Name(%s)" % repr(name)
self.names = {}
def visitSubscript(self, node, *args):
if repr(node.expr) == self.name:
try:
name = _evaluate(node.subs[0])
except:
name = str(node.subs[0])
self.names[name] = 1
def visitCallFunc(self, node, *args):
try:
from inspect import getsource, getargspec
from compiler import parse, walk
func = _evaluate(node.node)
pos = map(repr, node.args).index(self.name)
src, arg = getsource(func), getargspec(func)[0]
tmp, self.name = self.name, "Name(%s)" % repr(arg[pos])
walk(parse(src), self)
self.name = tmp
except Exception, e:
print str(e)

if __name__ == "__main__":

from inspect import getsource
from compiler import parse, walk

src = getsource(foo)
mod = parse(src)
visitor = RecursiveVisitor("kw")
walk(mod, visitor)
print visitor.names.keys()

-- black magic ends here, ouch --

Once again, I'm in total awe at the possibilities python offers us lowly
users. Unfortunately we're all the same: you give us a finger, we want the
whole arm! I want this method to generalize to method calls as in

class baz:

def foo(self, arg_dict):
return arg_dict["one"] + self.bar(arg_dict)

def bar(self, other_dict):
return other_dict["two"] + other_dict[True and "three" or "four"]

It shouldn't be all that hard. My problem is the lookup of 'self.bar'. In
the AST it looks something like

CallFunc(Getattr(Name('self'), 'bar'), [Name('arg_dict')], None, None)

How do I find the corresponding function? Anybody feels like wrapping
their head on this?

Cheers,

Jan Burgy
Jul 18 '05 #1
2 1983

"Jan Burgy" <jb****@hotmail.com> wrote in message
news:80**************************@posting.google.c om...
return arg_dict["one"] + self.bar(arg_dict) It shouldn't be all that hard. My problem is the lookup of 'self.bar'. In
the AST it looks something like

CallFunc(Getattr(Name('self'), 'bar'), [Name('arg_dict')], None, None)
How do I find the corresponding function?


The lookup returns a bound method, whose structure is an implementation
detail.

Completely untested suggestion, possibly not correct or possible:

Write an implementation-specific unwrap function. Then either augment the
source code

..... + unwrap(self.bar)(self, arg_dict) # self now an explict arg

or the analysis code to do the unwrapping. For the latter, recognize
Name('self') and replace the result of Getattr(Name('self'), 'bar') with
unwrap(thereof) and, again, augment the arg list.

Terry J. Reedy

Jul 18 '05 #2
"Terry Reedy" <tj*****@udel.edu> wrote in message news:<ma**************************************@pyt hon.org>...
"Jan Burgy" <jb****@hotmail.com> wrote in message
news:80**************************@posting.google.c om...
return arg_dict["one"] + self.bar(arg_dict)

It shouldn't be all that hard. My problem is the lookup of 'self.bar'. In
the AST it looks something like

CallFunc(Getattr(Name('self'), 'bar'), [Name('arg_dict')], None,

None)

How do I find the corresponding function?


The lookup returns a bound method, whose structure is an implementation
detail.

Completely untested suggestion, possibly not correct or possible:

Write an implementation-specific unwrap function. Then either augment the
source code

.... + unwrap(self.bar)(self, arg_dict) # self now an explict arg

or the analysis code to do the unwrapping. For the latter, recognize
Name('self') and replace the result of Getattr(Name('self'), 'bar') with
unwrap(thereof) and, again, augment the arg list.

Terry J. Reedy


Hi Terry,

Thank you very much for your suggestion. I am not sure how it would
help though. What is preventing me now from already replacing the
result of Getattr(Name('self'), 'bar') with what I need? That's
precisely my problem however. Because Python is a dynamically typed
language, I have no information about 'self' at compile-time. Short of
saving context information while I am visiting the parse tree, I don't
know how to tell what class I am in.

Would-Meta-Classes-help-me-in-this-case-ly yours?

Jan
Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: dan | last post by:
without stirring the pot too much -- could someone please point me to whatever documentation exists on the philosophy, semantics, and practical implications of how Python implements the...
0
by: Michael Spencer | last post by:
Wow - Alex Martelli's 'Black Magic' Pycon notes http://www.python.org/pycon/2005/papers/36/pyc05_bla_dp.pdf include this gem: > Functions 'r descriptors > def adder(x, y): return x + y > ...
2
by: Alex | last post by:
Entering the following in the Python shell yields >>> help(dict.copy) Help on method_descriptor: copy(...) D.copy() -> a shallow copy of D >>>
2
by: stewart.midwinter | last post by:
I've been experimenting with the python zipfile module, and have run into a snag. I'm able to create a new zipfile with the module's ZipFile class and add files to it. After closing the file,...
0
by: KronicDeth | last post by:
I have a package of python modules deployed on an NFS mount on my network that I use for sysadmin tools. Since some of the machines use different versions of python I've only put the .py files in...
3
by: Samuel | last post by:
Hi, How can I determine the type of a file from "magic bytes", similar to what the "file" command on unix does? I found http://docs.python.org/lib/module-mimetypes.html but this only seems...
16
by: per9000 | last post by:
Hi, I recently started working a lot more in python than I have done in the past. And I discovered something that totally removed the pretty pink clouds of beautifulness that had surrounded my...
9
by: Larry Hale | last post by:
I've heard tell of a Python binding for libmagic (file(1) *nixy command; see http://darwinsys.com/file/). Generally, has anybody built this and worked with it under Windows? The only thing I've...
3
by: Larry Hale | last post by:
Thank you, again, Michael, for all your help many months ago. I *FINALLY* got a HowTo done up; please see http://wiki.python.org/moin/HowTo/FileMagic I've also emailed Mr. Hupp to see if he'll...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.