Returning a value from code string

[Kirk McDonald]

Say I have a database containing chunks of Python code. I already have a
way to easily load, edit, and save them. What is slightly baffling to me
is how I can effectively pass this code some arguments, run it, and
somehow get a return value.

(Acutally, in the process of writing this post, I figured out a pretty
good way of doing it. Here's the rest of my post and the solution I came
up with. Enjoy my thought process!)

Right now I'm thinking of just defining a standard function name, and
using that as the entry/exit point for the DB-stored code. Is this
reasonable? Is there some more Pythonic way?

I'll probably compile the code and store the bytecode to the database
(in addition to the plain text) for faster execution time, as well. It
shouldn't care whether the stored code is a string or a code object.

What if a stored chunk of code doesn't need to return anything? It would
be more convenient to just execute the code without the standard
function. Also, it'd be nice to wrap this in a class:

# module test.py
class Codeobj:
def __init__(self, code=None):
if code: self.code = code
else: self.code = ""

def __call__(self, args=None, **kwargs):
# We can pass in a name/value dictionary
if args: kwargs.update(args)
exec self.code
# We don't need a standard function if we're not returning
# anything
if locals().has_key('std_func_name'):
return std_func_name(**kwargs)

if __name__ == "__main__":
# Load code from the DB
double = """\
def std_func_name(n):
return n*2"""

addthree = """\
def std_func_name(n):
return n+3"""

noreturn = "print 'arg = %s' % kwargs['arg']"

a = Codeobj(double)
b = Codeobj(addthree)
c = Codeobj(noreturn)

# Calling with name/value dictionary
print a(args={'n':5})
# Calling with specific named argument
print b(n=4)
c(arg='foo')
# EOF

$ python test.py
10
7
arg = foo

If I wanted to simplify the 'noreturn' example (well, if there were more
lines of code this might make it simpler), I could have defined it like
this:

noreturn = """\
locals().update(kwargs)
print 'arg = %s' % arg"""

-Kirk McDonald

[Steven D'Aprano]

On Fri, 27 Jan 2006 20:33:53 -0800, Kirk McDonald wrote:

Say I have a database containing chunks of Python code. I already have a
way to easily load, edit, and save them.
Why?

What is slightly baffling to me
is how I can effectively pass this code some arguments, run it, and
somehow get a return value.

You already know about exec. Do you know about eval?

result = eval(some_piece_of_text)

I hope the contents of the database are trusted, because if the code is
coming from an untrusted source, well, U R pwn3d.

Seems like you are jumping through a lot of hoops for very little benefit.
What am I missing?
--
Steven.

[Kirk McDonald]

Steven D'Aprano wrote:

On Fri, 27 Jan 2006 20:33:53 -0800, Kirk McDonald wrote:

Say I have a database containing chunks of Python code. I already
have a way to easily load, edit, and save them.

Why?

I am implementing an Everything Engine-like system (see everything2.com
and everydevel.com) in Python. (The original is in Perl, and by all
accounts is some pretty ugly code.) The Everything idiom is that
everything in the engine is a 'node.' A node is a discrete unit that can
be loaded from and saved to the database. A node can be asked to display
itself in the page, it has an owner (and users are themselves nodes),
and so on.

One kind of node is a Superdoc, which is a document that mixes HTML with
code. (I have simply implemented these as mod_python PSP documents,
which is all hunky-dory.) Superdocs comprise essentially all of the
important bits of the site. Another kind of node (I'm still deciding
whether to call them Codenodes or Opcodes or maybe Pynodes) is a chunk
of code that can be asked to run itself, and which can be edited, on the
fly, from within the website. Thus, one can both alter the functionality
of the site, and add functionality, from the site itself (so long as you
have the user permissions to do so).

-Kirk McDonald

[Fredrik Lundh]

Kirk McDonald wrote:

def __call__(self, args=None, **kwargs):
# We can pass in a name/value dictionary
if args: kwargs.update(args)
exec self.code
# We don't need a standard function if we're not returning
# anything
if locals().has_key('std_func_name'):
return std_func_name(**kwargs)

executing the code in a custom namespace is a lot cleaner:

ns = {}
ns["os"] = os # insert "preimported" values
ns["app"] = app # insert "preloaded" objects
exec self.code in ns
try:
func = ns["std_func_name"]
except KeyError:
pass
else:
func(**kwargs)

instead of using a standard function, you can let the code
objects talk to the application via a preloaded object or an
interface module. e.g. instead of

def std_func_name(app, args):
app.something()

your code object could simply become

app.something()

or, if std_func_name represents a specific event:

def myfunc(args):
app.something()

app.register("event", myfunc)

(in the latter case, the event handler don't have to know
about Codeobj instances; all event callbacks are ordinary
callables)

</F>

[Kirk McDonald]

Fredrik Lundh wrote:

executing the code in a custom namespace is a lot cleaner:

ns = {}
ns["os"] = os # insert "preimported" values
ns["app"] = app # insert "preloaded" objects
exec self.code in ns
try:
func = ns["std_func_name"]
except KeyError:
pass
else:
func(**kwargs)

instead of using a standard function, you can let the code
objects talk to the application via a preloaded object or an
interface module. e.g. instead of

def std_func_name(app, args):
app.something()

your code object could simply become

app.something()

These code objects should all return an instance of a certain class (or
one of its descendants). In another branch of this topic, I described
roughly what this is for. So, without going into too much detail, I'll
just repeat that this is for a website backend. Whenever an HTML form is
submitted, a hidden "op" element (short for operation) in the form is
checked. This value will send the rest of the form data on to the code
object it represents. (If you've read my other post: These code objects
are nodes, the value for the "op" element is the ID number of one of
them, or possibly one of a few mnemonics for some of the more important
ops.)

So, taking another swipe at it with your suggestions:

def __call__(self, args=None, **kwargs):
ns = {}
ns.update(kwargs)
ns["args"] = args
ns["retval"] = None
exec self.code in ns
try:
func = ns["std_func_name"]
except KeyError:
return ns["retval"]
else:
# Already passed kwargs into the namespace
return func()

Then, a code object can either return a value by using whatever happens
to be in 'retval' when it's done executing, or by defining a
'std_func_name' function and using its return value.

So the interface for these Codenodes needs to be able to accept
arguments as a name/value dictionary corresponding to the elements in
the HTML form (the 'args' argument in my example above). This dictionary
is passed on to the code object. The kwargs are a few important
variables (not supplied by the form) that all Codenodes need to know,
such as the current user and maybe a database cursor.

Each Codenode knows how to deal with the input from its associated form.
(Or forms! There is, for example, a generalized "commit" Codenode that
knows how to update the state of *any* node if a form follows a certain
template.) When it is done, it needs to then return whichever node
should be displayed next. If the operation being attempted violates the
user's permissions, it might return the "access denied" node. (Actually,
in that case it will raise an AccessDenied exception and expect
something down the line to deal with it, so I can define what the
"access denied" node is in a central place, but you get the idea.) If
the operation resulted in a new node being created, it might return the
new node. &c.

Sorry if I tend to ramble, heh. :-)

-Kirk McDonald

[Max]

Kirk McDonald wrote:

Another kind of node (I'm still deciding
whether to call them Codenodes or Opcodes or maybe Pynodes) is a chunk
of code that can be asked to run itself, and which can be edited, on the
fly, from within the website. Thus, one can both alter the functionality
of the site, and add functionality, from the site itself (so long as you
have the user permissions to do so).

As Steven said, "U R pwn3d". 1f you d0n't sp3a|< l33t (if you don't
speak leet), that means you are screaming "hack me, use me to launch
attacks on other computers, and than attack my computer". Unless you
have some revolutionary ideas in code-security analysis. In which case
you can a lot more money than from implementing Everything2 in python.
-Kirk McDonald

--Max

[Kirk McDonald]

Max wrote:

Kirk McDonald wrote:

Another kind of node (I'm still deciding
whether to call them Codenodes or Opcodes or maybe Pynodes) is a chunk
of code that can be asked to run itself, and which can be edited, on
the fly, from within the website. Thus, one can both alter the
functionality of the site, and add functionality, from the site itself
(so long as you have the user permissions to do so).

As Steven said, "U R pwn3d". 1f you d0n't sp3a|< l33t (if you don't
speak leet), that means you are screaming "hack me, use me to launch
attacks on other computers, and than attack my computer". Unless you
have some revolutionary ideas in code-security analysis. In which case
you can a lot more money than from implementing Everything2 in python.

--Max

Heavens! Normal users can't edit code! They won't even see it! I'm not a
*total* moron. The only thing users will be able to enter is some
simplified HTML. This is a convenience feature for the (trusted) admins
of the site. There are some simple permission features built into the
API. Every database-altering API call takes the current user as an
argument, and if they're not allowed, it tells them to get bent.

Everything2 does this more or less the same way, and they've had few
issues in the seven or so years they've been operating.

-Kirk McDonald

[Fried Egg]

>From above:

I hope the contents of the database are trusted, because if the code is
coming from an untrusted source, well, U R pwn3d.

Seems like you are jumping through a lot of hoops for very little
benefit.
What am I missing?

<END>

Why does anyone care about "why" people do things when they ask a
specific technical question on a newsgroup? Maybe op is risking his
server (who cares) or maybe he is just trying to explore an idea
(again, who cares). It almost seems like bad etiquette, really. It
also seems somehow different than asking questions that elicit
technical context.

Sorry to rant about etiquette. I just have a problem similar to the
original poster's, and it pissed me off when I wrote to the newsgroup
asking "how do I do x,y,z?" and somebody writes back saying "you really
shouldn't want to do x,y,z..." when they really haven't a clue. This
exchange has been a lot better than that, probably because of my poor
explanation in my question.

[Alex Martelli]

Fried Egg <we**********@gmail.com> wrote:
...

Why does anyone care about "why" people do things when they ask a
specific technical question on a newsgroup? Maybe op is risking his

Because experienced techies have learned that people (including other
techies) often ask the wrong "specific technical question", having
previously made some badly-founded assumptions and premature preliminary
design decisions; by taking a step back, and getting the "why", it's
therefore quite often possible to help the querant come up with a much
better overall solution.

The classic example is right at the start of Bentley's classic "Writing
Efficient Programs" (or maybe "Programming Pearls" -- been a while since
I read either!-), where another programmer asks him how to invoke the
system sort from within a program, and Bentley at first wastes a lot of
time and energy answering the exact question that was asked -- but
finally steps back, DOES ask the crucial question, "why do you want to
know?", and helps the querant find a solution which runs rings around
anything the querant might have come up with by shelling out to a system
sort. Many experienced techies are familiar with Bentley's
masterpieces, and the lesson appears to have sunk in -- or it may well
have been rediscovered independently many, many times, of course.

It's unfortunate that you appear not to understand or accept this, but I
hope nobody will change their helpful behavor and motivation as a
consequence of your complaint. It's only by relentlessly asking "why",
that very experienced developers can *most* help others.
Alex

[Peter Hansen]

Fried Egg wrote:

From above: Seems like you are jumping through a lot of hoops for very little
benefit.
What am I missing?
<END>

Why does anyone care about "why" people do things when they ask a
specific technical question on a newsgroup? Maybe op is risking his
server (who cares) or maybe he is just trying to explore an idea
(again, who cares). It almost seems like bad etiquette, really. It
also seems somehow different than asking questions that elicit
technical context.

To be blunt, you're wrong. Though some of us have trouble posting
"why?" responses that never sound rude, it is not only a good idea to
understand why, it's *necessary*. In fact, only by understanding why
can one really understand the "technical context". This is a basic
tenet of good requirements engineering.

Among other good reasons to ask why, there are the following:

1. Experience in this newsgroup shows that a large percentage of people
asking questions have a poor understanding of their own requirements, or
are making the mistake of defining their problem in terms of a desired
solution rather than merely specifying requirements and allowing the
experts to choose the most appropriate solution. An example of this is
the frequent "How can I use package X to do such-and-such?" where the
ultimate best answer turns out to be "don't use package X, you don't
need it for what it turns out you're trying to do". Yes, sometimes the
OP specifically wanted to use X to solve the problem, to learn more
about X, but then that's just when asking "why?" is most critical. They
usually don't tell us when that's the case, so we have to ask to be sure.

2. Because of different linguistic and cultural backgrounds in this
international newsgroup, or the inherent ambiguities in human language,
a poster sometimes misuses a word or phrase, or entirely misunderstands
something he's read, and if we don't dig down to the "why?" we risk
providing an answer which answers the letter of his question but
entirely misses the spirit, and ultimately is a disservice.

3. People posting here have a wide range of experience, whether with
Python, with programming, or with computers in general. Sometimes it's
not at all clear how experienced a poster is, and if a given answer
doesn't take into account the possibility that the poster is such a
rookie that he might be completely off the mark about how to do
something, or even whether it's a good idea to do something, then that
answer might be more harmful than helpful.

In the specific case in question, Kirk's first post gave little or no
hint about how much he knew about security issues and, by asking "why?",
with a single word Steven hoped to learn enough to say things like "oh,
there's a package designed to do that which you can just plug in" or
"oh, it's clear you are totally new to issues of network security and
are about to put your foot right in it, so here's some background to
save you from yourself", or whatever...
Sorry to rant about etiquette. I just have a problem similar to the
original poster's, and it pissed me off when I wrote to the newsgroup
asking "how do I do x,y,z?" and somebody writes back saying "you really
shouldn't want to do x,y,z..." when they really haven't a clue. This
exchange has been a lot better than that, probably because of my poor
explanation in my question.

Being pissed off when someone trying to help you responds in a manner
you consider to be overstepping some unspoken bounds is really your
problem, not that of the respondent, and you ought to learn to deal with
it given, the anarchy that is Usenet. While it's arguable whether
bluntly asking "why?" violates some kind of "etiquette", it most
definitely does *not* violate "netiquette", and actually follows it very
closely. IMHO.

YMMV, but ultimately it's your problem if you don't think people should
ask why. If it really concerns you, prepend your posts with "I believe
I have good reasons for doing the things I'm doing here and kindly ask
that potential respondents limit their responses to directly addressing
the technical issues I raised and not asking "why?" I'm doing this.
Thank you in advance."

If you were to do that, it would largely (but not entirely) eliminate
that kind of reply, and would save some of us the trouble of reading the
rest of your post.

-Peter

[Kirk McDonald]

Peter Hansen wrote:

In the specific case in question, Kirk's first post gave little or no
hint about how much he knew about security issues and, by asking "why?",
with a single word Steven hoped to learn enough to say things like "oh,
there's a package designed to do that which you can just plug in" or
"oh, it's clear you are totally new to issues of network security and
are about to put your foot right in it, so here's some background to
save you from yourself", or whatever...

Heh heh. For what it's worth, I did have a perfectly good reason for
trying to do what might otherwise be a suicidally insane idea. On the
other hand, I've been re-evaluating my needs, and I'll probably be
hardcoding the more important bits of code that I was going to store in
the database (in that pulling these things from the database whenever I
need to use them is remarkably slow compared to letting mod_python cache
them, and these particular functions are unlikely to ever change). If I
do need a way to have dynamic code in the database, I've added a thing
that just reflects POST data back to the database-stored PSP document in
which the form was found (if the form wants it to), letting the PSP
document deal with it. That should be adequate.

In short, I learned something about exec and namespaces that I'm not
actually going to use. Hooray!

-Kirk McDonald

[Steven D'Aprano]

Fried Egg wrote:

From above:
I hope the contents of the database are trusted, because if the code is
coming from an untrusted source, well, U R pwn3d.

Seems like you are jumping through a lot of hoops for very little
benefit.
What am I missing?

<END>

Why does anyone care about "why" people do things when they ask a
specific technical question on a newsgroup? Maybe op is risking his
server (who cares) or maybe he is just trying to explore an idea
(again, who cares). It almost seems like bad etiquette, really. It
also seems somehow different than asking questions that elicit
technical context.

Firstly:-

Because people have an ethical obligation not to give
bad advice.

You might be prepared to assume that anonymous people
on the Net are all responsible, intelligent,
clear-thinking, sufficiently cautious adults who know
what they are doing, but I don't make that assumption.
If their post indicates they know what they are doing,
I respect that and am more than happy to tell them how
to shoot themselves in the foot. If their post
indicates that they haven't thought the question
through, or worse, aren't even aware that there is a
question to think through, then I'll behave differently.

If somebody writes to the newsgroup "Hi, I need help
writing a Caesar shift cipher so I can encrypt some
data" it would be sheer irresponsibility to answer
their technical question without pointing out that a
Caesar shift cipher is no more secure than rot13 (in
fact, rot13 is a specific form of that cipher). So ask
"Why?" -- if the answer is "I'm storing confidential
medical records in a database", then you can gently
apply the cluebat and, if you are lucky, protect YOUR
medical records from being stolen from some clueless
database vendor. If the answer is "I'm doing it to
obfuscate some data in a game, I know this is extremely
weak encryption", then that's a different ballgame.
Secondly:-

Because one of the most common errors on Usenet and the
Internet is people asking the wrong question because of
misunderstanding or false information. A recent example
on this newsgroup: somebody had assumed, or been told,
that Python lists were linked lists, and assuming that
mylist[n] was a slow procedure because the linked list
had to be walked, was wasting time trying to some up
with some other solution. If we had merely answered
their technical question, he would have solved a
non-problem, learnt nothing, and ended up with slow
inefficient code.

His real problem wasn't "How do I this...?". His real
problem was that he was labouring under false
apprehensions, and by asking "Why do you want to do
this?", people solved his real problem.
Sorry to rant about etiquette. I just have a problem similar to the
original poster's, and it pissed me off when I wrote to the newsgroup
asking "how do I do x,y,z?" and somebody writes back saying "you really
shouldn't want to do x,y,z..." when they really haven't a clue.

But the million dollar question is, after you had
explained what you wanted to accomplish (rather than
how you thought it should be accomplished), did people
agree that x,y,z was the right way to go about it?

--
Steven.

[Fried Egg]

<SNIP> All the well reasoned responses about asking possibly out of
context why questions </SNIP>

Well, I won't be so impatient the next time someone misconstrues what I
thought was a simple question and lectures me about SQL injection or
whatever.

Additionally, if I think something might be considered a little nuts, I
will add a line to the effect of "This is crazy, I just want to figure
out how to do it, you don't need to lecture me about SQL injection".

I hadn't considered the viewpoint of those on the newsgroup who
faithfully answer questions day in and day out, who are basically
volunteering as tutors to the community, and who see the conversations
in a very different light than I do. Thanks to them for their time,
and I appreciate their (as usual) excellently written responses.

[Steve Holden]

Fried Egg wrote:

<SNIP> All the well reasoned responses about asking possibly out of
context why questions </SNIP>

Well, I won't be so impatient the next time someone misconstrues what I
thought was a simple question and lectures me about SQL injection or
whatever.

Additionally, if I think something might be considered a little nuts, I
will add a line to the effect of "This is crazy, I just want to figure
out how to do it, you don't need to lecture me about SQL injection".

I hadn't considered the viewpoint of those on the newsgroup who
faithfully answer questions day in and day out, who are basically
volunteering as tutors to the community, and who see the conversations
in a very different light than I do. Thanks to them for their time,
and I appreciate their (as usual) excellently written responses.

Yup, unsung heroes all! :-)

perhaps-slightly-sung-ly y'rs - steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/