Re: __builtins__ magic behavior

[Gabriel Genellina]

En Sun, 07 Sep 2008 14:00:48 -0300, Patrick Maupin <pm*****@gmail.comescribió:

__builtins__ in 2.5.2 doesn't seem to behave like I remember it did
the last time I did some custom stuff with it, a very long time ago.

This isn't surprising, because of ongoing optimization, but it's hard
to google for '__builtins__' so I didn't really find any documentation
on the current CPython behavior, which in some cases seems quite
strange to me.

The documentation node at http://docs.python.org/ref/naming.html has a
"here be dragons" note on __builtins__, which is nice as far as it
goes, but doesn't provide any help for people wanting to abuse the
current CPython implementation. Additionally, the sentence
immediately above the note, "__builtins__ can be set to a user-created
dictionary to create a weak form of restricted execution" not only
contradicts the note, but in some cases appears not to be true.

When the code below is run, either standalone or by importing it, the
builtins used are different depending on whether exec is used with
globals() or with a copy of globals(). Any explanations for this
behavior would be much appreciated.

Python takes some shortcuts when dealing with builtins. I'll just describe what happens (I won't say whether it is "right" or "wrong").
The exec statement, when given a string source, compiles it and eventually calls PyEval_EvalCodeEx, which creates a new frame using PyFrame_New and finally executes it.
A frame object contains a pointer to the previous frame, the code to be executed, a pointer to the current globals *and* a separate pointer to the current builtins.
Inside PyFrame_New, there is a shortcut: if the new frame and the previous one share the same globals, then the previous builtins are copied into the new frame. Only if the globals differ the builtins are searched in globals. From frameobject.c: /* If we share the globals, we share the builtins. Save a lookup and a call. */
It is this assumption that fails in your code.

If you want to execute some code with modified builtins, do not change __builtins__ in the *calling* code, but in the globals that you pass to the exec call. That appears to be the most logical approach, and the way the developers appear to have expected.

--
Gabriel Genellina

[Patrick Maupin]

On Sep 7, 2:50*pm, "Gabriel Genellina" <[e-mail]:

Inside PyFrame_New, there is a shortcut: if the new frame and
the previous one share the same globals, then the previous
builtins are copied into the new frame. Only if the globals
differ the builtins are searched in globals. From frameobject.c:
/* If we share the globals, we share the builtins. Save a
lookup and a call. */

That was exactly my problem. Thank you for the cogent explanation.

If you want to execute some code with modified builtins, do not
change __builtins__ in the *calling* code, but in the globals that
you pass to the exec call. That appears to be the most logical
approach, and the way the developers appear to have expected.

Actually, my use-case is a tiny bit different than this. I am
dynamically creating module objects, and was going nuts trying
to comprehend the inner workings of behind-the-scenes __builtins__
magic in the module object, not realizing that the magic was
actually in the execution frames.

Thanks,
Pat

[Steven D'Aprano]

Gabriel, could I please ask you to configure your news-reader software or
editor to limit the length of each line of your posts to 70 characters
wide, as per the common standard for email and Usenet? Your lines are
significantly longer than that, including one single line which is 325
characters wide. That forces a lot of backwards and forwards scrolling to
read your posts.

Thank you.

--
Steven

[Gabriel Genellina]

En Sun, 07 Sep 2008 19:30:07 -0300, Steven D'Aprano
<st***@remove-this-cybersource.com.auescribió:

Gabriel, could I please ask you to configure your news-reader software or
editor to limit the length of each line of your posts to 70 characters
wide, as per the common standard for email and Usenet? Your lines are
significantly longer than that, including one single line which is 325
characters wide. That forces a lot of backwards and forwards scrolling to
read your posts.

Fixed, thanks for pointing it out. Opera wraps long lines automatically so
I didn't notice the problem myself.
I can't remember when, or why, I altered the setting (if *I* altered it at
all!)

--
Gabriel Genellina