473,887 Members | 2,283 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Proposed PEP: Treating Builtins as Constants in the Standard Library

Comments are invited on the following proposed PEP.
Raymond Hettinger

-------------------------------------------------------

PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com >
Status: Pre-PEP
Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.
Motivation
==========

The library contains code such as _len=len which is intended to create
fast local references instead of a slower global lookups. Though
necessary for performance, these constructions clutter the code and
are usually incomplete (missing many opportunities).

If the proposal is adopted, those constructions could be eliminated
from the code base and at the same time improve upon their results
in terms of performance.

There are currently over a hundred instances of ``while 1`` in the
library. They were not replaced with the more readable ``while True``
because of performance reasons (the compiler cannot eliminate the
test because True is not known to always be a constant). Conversion
of True to a constant will clarify the code while retaining performance.

Many other basic Python operations run much slower because of global
lookups. In a try/except statement, the matching exception must be
re-looked up on every call (whether or not the exception is raised).
Similarly, simple identity tests such as ``while input is not None``
require the None variable to be re-looked up on every pass.
Builtin lookups are especially egregious because the enclosing
global scope must be checked first. These lookup chains devour cache
space that is best used elsewhere.

In short, if the proposal is adopted, the code will become cleaner
and performance will improve across the board.
Proposal
========

Add an internal module called _bind.py which contains two functions,
bind_builtins() and bind_builtins_a nd_globals().

For each module in the standard library, add a pair of lines near
the near of the code::

import _bind, sys
_bind.bind_buil tins(sys.module s[__name__])

In a few select cases (such as sre_compile) where the module variables
are just a collection of constants, the globals will also be converted
to constants using bind_builtins_a nd_globals(). This will not be
applied anywhere there would be cause for a user to externally
change the value of one of the module variables.
Question and Answers
=============== =====

Q. Will this make everyone divert their attention to optimization issues?

A. No, it is internal only (not user visible). Also, it reduces the need
to think about optimization because it is done automatically.
Q. What if the module defines a variable shadowing a builtin?

A. This does happen. For instance, True can be redefined at the module
level as `True = (1==1)`. The sample implementation below detects the
shadowing and leaves the global lookup unchanged.
Q. How do you know this works?

A. I implemented it, applied it to every module in library, and ran the
test suite without exception.
Q. Are you the first person to recognize that most global lookups are for
values that never change?

A. No, this has long been known. Skip provides an especially eloquent
explanation in [1]_.
Q. Why not make this a public module so that users can optimize builtins
or builtins and globals in their own code?

A. The ASPN recipe [2]_ is provided for this purpose. It takes a modicum
of skill to use correctly and not inadvertently tranform a non-constant.
Q. What if I have an overpowering urge to import a library module,
write a new shadowing builtin, and insert it into the module's namespace?

A. Your problems are less acute than with a module written in C
because you can always comment out the automatic binding.
Q. More realistically, what if I want to replace the builtins module and
supply my own implementations ?

A. Either do this before importing a module, or just reload the module,
or disable _bind.py.
Q. How susceptible is this module to changes in Python's byte coding?

A. It imports opcode.py to protect against renumbering. Also, it uses
LOAD_CONST and LOAD_GLOBAL which are fundamental and have been around
forever. That notwithstanding , the coding scheme could change
dramatically and this implementation would have to change along with
modules like `dis` which also rely on the current coding scheme.

Sample Implementation
=============== ======

Here is a sample implementation for _bind.py::

from types import ClassType, FunctionType
from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG
LOAD_GLOBAL, LOAD_CONST = opmap['LOAD_GLOBAL'], opmap['LOAD_CONST']

__all__ = ['bind_builtins' , 'bind_builtins_ and_globals']

def bind(f, builtin_only=Tr ue):
""" Return a new function with global references converted to constants.

If builtin_only is True (the default), will convert all builtin
references unless they have been overridden in the function's globals.
If builtin_only is False, will also convert other module variable
references into constants.

"""
import __builtin__
env = vars(__builtin_ _).copy()
if builtin_only:
stoplist = f.func_globals # do not replace overridden builtins
else:
stoplist = {}
env.update(f.fu nc_globals)

co = f.func_code
newcode = map(ord, co.co_code)
newconsts = list(co.co_cons ts)
codelen = len(newcode)

i = 0
while i < codelen:
opcode = newcode[i]
if opcode == EXTENDED_ARG:
return f # handle only the simple cases
if opcode == LOAD_GLOBAL:
oparg = newcode[i+1] + (newcode[i+2] << 8)
name = co.co_names[oparg]
if name in env and name not in stoplist:
value = env[name]
try:
pos = newconsts.index (value)
except ValueError:
pos = len(newconsts)
newconsts.appen d(value)
newcode[i] = LOAD_CONST
newcode[i+1] = pos & 0xFF
newcode[i+2] = pos >> 8
i += 1
if opcode >= HAVE_ARGUMENT:
i += 2

codestr = ''.join(map(chr , newcode))
if codestr == co.co_code:
return f # no changes were found
codeobj = type(co)(co.co_ argcount, co.co_nlocals, co.co_stacksize ,
co.co_flags, codestr, tuple(newconsts ), co.co_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstline no, co.co_lnotab, co.co_freevars,
co.co_cellvars)
return type(f)(codeobj , f.func_globals, f.func_name, f.func_defaults ,
f.func_closure)

def bind_all(module _or_class, builtin_only=Tr ue):
"""Recursiv ely apply bind() to functions in a module or class.

Use as the last line of the module (after everything is defined, but
before test code or lines defining shorthand references to functions.

If builtin_only is True (the default), will convert all builtin
references unless they have been overridden in the function's globals.

If builtin_only is False, will also convert other module variable
references into constants. This should only be used in modules
where the values of the module variables are truly constant.

"""
for k, v in vars(module_or_ class).items():
if type(v) is FunctionType:
newv = bind(v)
setattr(module_ or_class, k, newv)
elif type(v) in (type, ClassType):
bind_all(v, builtin_only)

import sys
bind_all(sys.mo dules[__name__], False) # Binder, bind thyself!

# Self-documenting API (eliminates the builtin_only switch)
bind_builtins = bind_all
bind_builtins_a nd_globals = lambda m: bind_all(m, False)
The final code should have some minor embellishments:

* Automatic detection of a non-CPython environment that does not have
bytecodes. In that situation, the bind functions would simply
return the original function unchanged. This assures that the two
line additions to library modules do not impact other implementations .

* Add a few lines to handle reading and writing of EXTENDED_ARG. While
this almost never arises in practice, it is best to have the code as
adaptable as possible.
References
==========

... [1] Optimizing Global Variable/Attribute Access
http://www.python.org/peps/pep-0266.html

... [2] ASPN Recipe for a non-private implementation
http://aspn.activestate.com/ASPN/Coo.../Recipe/277940

Copyright
=========

This document has been placed in the public domain.

...
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
Jul 18 '05 #1
8 1878

"Raymond Hettinger" <py****@rcn.com > wrote in message
news:5d******** *************** ***@posting.goo gle.com...
Comments are invited on the following proposed PEP.
Raymond Hettinger


It certainly looks interesting. What's the possibility
of also eliminating the extra function call on global
functions like len() at the same time?

John Roth
Jul 18 '05 #2
Raymond Hettinger wrote:
PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com >
Status: Pre-PEP
Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.



Two things:

1. The new _bind module needs to have a more specific (and
appropriate) name than _bind. The function in the module could
still be named bind, but the module itself should be named
something like

_optimize_globa ls_as_consts_ve ry_unsafe

(Ok, but you get the idea.)

2. The module being optimized should have a special attribute, say
__variable__, that identifies any globals that should not be
optimized in this way:

__variable__ = ['global_that_mi ght_change']
Other than that, +1. I'm definitely in favor of optimization hacks
whenever it doesn't intrude too much on the code. It would be nice if
it were safer.
--
CARL BANKS http://www.aerojockey.com/software
"If you believe in yourself, drink your school, stay on drugs, and
don't do milk, you can get work."
-- Parody of Mr. T from a Robert Smigel Cartoon
Jul 18 '05 #3
rh> Q. Why not make this a public module so that users can optimize
rh> builtins or builtins and globals in their own code?

rh> A. The ASPN recipe [2]_ is provided for this purpose. It takes a
rh> modicum of skill to use correctly and not inadvertently tranform
rh> a non-constant.

I thought one of the basic tenets of Python was "we're all adults here".
Programmers have unfettered access to the new module with which they can
wreak havoc. I don't see this proposal as somehow any more dangerous.

Skip
Jul 18 '05 #4
Raymond Hettinger wrote:
Comments are invited on the following proposed PEP.
Raymond Hettinger

-------------------------------------------------------

PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com >
Status: Pre-PEP
Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.


Is it really a good thing to do it manually?
Shouldn't be done automatically by a simple analyser?

--
Yermat

Jul 18 '05 #5
Yermat wrote:
Raymond Hettinger wrote:
PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com >

Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.


Is it really a good thing to do it manually?
Shouldn't be done automatically by a simple analyser?


Not requiring that it be done manually, or at least not providing
a simple means of disabling it, would prevent some kinds of
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.

-Peter
Jul 18 '05 #6
On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <pe***@engcorp. com>
wrote:

[snip]
....
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.


Isn't this the use case for AOP you were looking for?
--dang
Jul 18 '05 #7
Daniel 'Dang' Griffith wrote:
On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <pe***@engcorp. com>
wrote:

[snip]
...
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.

Isn't this the use case for AOP you were looking for?
--dang


As you might imagine from my previous postings, I have no idea.
I don't understand well enough what AOP really is to say,
obviously (given my requests for compelling use cases so I can
understand why I would even want to investigate it).

Can you describe how AOP would solve this problem? As I
understand it so far, it has nothing to do with mock objects...

-Peter
Jul 18 '05 #8
On Tue, 20 Apr 2004 10:33:36 -0400, Peter Hansen <pe***@engcorp. com>
wrote:
Daniel 'Dang' Griffith wrote:
On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <pe***@engcorp. com>
wrote:
...
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.


Isn't this the use case for AOP you were looking for?
--dang


As you might imagine from my previous postings, I have no idea.
I don't understand well enough what AOP really is to say,
obviously (given my requests for compelling use cases so I can
understand why I would even want to investigate it).

Can you describe how AOP would solve this problem? As I
understand it so far, it has nothing to do with mock objects...


Probably not.
I'm not an "AOP person", but from what I understand, you could use
an AOP-enabled language to replace the real file system (or whatever)
objects with the mock objects. This is roughly analagous to the
standard logging or auditing examples, wherein a program is doing what
it needs to do, meanwhile a cross-cutting aspect intercepts certain
operations and performing additional, or different, operations.
--dang
Jul 18 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

59
4369
by: Raymond Hettinger | last post by:
Please comment on the new PEP for reverse iteration methods. Basically, the idea looks like this: for i in xrange(10).iter_backwards(): # 9,8,7,6,5,4,3,2,1,0 <do something with i> The HTML version is much more readable than the ReST version. See: http://www.python.org/peps/pep-0322.html
36
2209
by: Gonçalo Rodrigues | last post by:
I've lost the original thread on my news reader so I'm opening a new one. First, i like the name ireverse, much better then inreverse (eek!). Just a simple comment/question: - Why a builtin function? Why not just stuff it in the itertools module? The builtins is already fat as it is, making it fatter is not the way to go, IMHO. I'd like to remeber the copy module as a case
0
990
by: Raymond Hettinger | last post by:
Comments are invited on a new pep: http://www.python.org/peps/pep-0329.html Based on feedback received so far, there were several changes to the original draft: * The module is public
38
2612
by: Kevin Smith | last post by:
For what it's worth, I wrote the original PEP 318. I probably wasn't qualified, but I just wanted a nice simple way to declare class methods without having to repeat the function name. After submitting it to BDFL for approval, more work was needed and the discussion of PEP 318 on python-dev increased rapidly. It was evident that I was in over my head, so I asked more someone more experienced to take over. I guess others had bigger...
108
6505
by: Bryan Olson | last post by:
The Python slice type has one method 'indices', and reportedly: This method takes a single integer argument /length/ and computes information about the extended slice that the slice object would describe if applied to a sequence of length items. It returns a tuple of three integers; respectively these are the /start/ and /stop/ indices and the /step/ or stride length of the slice. Missing or out-of-bounds indices are handled in a manner...
77
3986
by: Ben Finney | last post by:
Howdy all, PEP 354: Enumerations in Python has been accepted as a draft PEP. The current version can be viewed online: <URL:http://www.python.org/peps/pep-0354.html> Here is the reStructuredText source as it is today. Please discuss it here so I can see what issues people may have.
4
2507
by: Tony Lownds | last post by:
(Note: PEPs in the 3xxx number range are intended for Python 3000) PEP: 3107 Title: Function Annotations Version: $Revision: 53169 $ Last-Modified: $Date: 2006-12-27 20:59:16 -0800 (Wed, 27 Dec 2006) $ Author: Collin Winter <collinw@gmail.com>, Tony Lownds <tony@lownds.com> Status: Draft Type: Standards Track
4
1679
by: dustin | last post by:
I've been hacking away on this PEP for a while, and there has been some related discussion on python-dev that went into the PEP: http://mail.python.org/pipermail/python-dev/2007-February/070921.html http://mail.python.org/pipermail/python-dev/2007-February/071155.html http://mail.python.org/pipermail/python-dev/2007-February/071181.html I'd love to have feedback on this PEP: - from a user's perspective (would you want to write...
399
13123
by: =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= | last post by:
PEP 1 specifies that PEP authors need to collect feedback from the community. As the author of PEP 3131, I'd like to encourage comments to the PEP included below, either here (comp.lang.python), or to python-3000@python.org In summary, this PEP proposes to allow non-ASCII letters as identifiers in Python. If the PEP is accepted, the following identifiers would also become valid as class, function, or variable names: Löffelstiel,...
0
9957
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9799
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11173
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10877
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9593
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7988
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6011
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4633
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4239
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.