473,371 Members | 1,753 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,371 software developers and data experts.

Proposed PEP: Treating Builtins as Constants in the Standard Library

Comments are invited on the following proposed PEP.
Raymond Hettinger

-------------------------------------------------------

PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com>
Status: Pre-PEP
Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.
Motivation
==========

The library contains code such as _len=len which is intended to create
fast local references instead of a slower global lookups. Though
necessary for performance, these constructions clutter the code and
are usually incomplete (missing many opportunities).

If the proposal is adopted, those constructions could be eliminated
from the code base and at the same time improve upon their results
in terms of performance.

There are currently over a hundred instances of ``while 1`` in the
library. They were not replaced with the more readable ``while True``
because of performance reasons (the compiler cannot eliminate the
test because True is not known to always be a constant). Conversion
of True to a constant will clarify the code while retaining performance.

Many other basic Python operations run much slower because of global
lookups. In a try/except statement, the matching exception must be
re-looked up on every call (whether or not the exception is raised).
Similarly, simple identity tests such as ``while input is not None``
require the None variable to be re-looked up on every pass.
Builtin lookups are especially egregious because the enclosing
global scope must be checked first. These lookup chains devour cache
space that is best used elsewhere.

In short, if the proposal is adopted, the code will become cleaner
and performance will improve across the board.
Proposal
========

Add an internal module called _bind.py which contains two functions,
bind_builtins() and bind_builtins_and_globals().

For each module in the standard library, add a pair of lines near
the near of the code::

import _bind, sys
_bind.bind_builtins(sys.modules[__name__])

In a few select cases (such as sre_compile) where the module variables
are just a collection of constants, the globals will also be converted
to constants using bind_builtins_and_globals(). This will not be
applied anywhere there would be cause for a user to externally
change the value of one of the module variables.
Question and Answers
====================

Q. Will this make everyone divert their attention to optimization issues?

A. No, it is internal only (not user visible). Also, it reduces the need
to think about optimization because it is done automatically.
Q. What if the module defines a variable shadowing a builtin?

A. This does happen. For instance, True can be redefined at the module
level as `True = (1==1)`. The sample implementation below detects the
shadowing and leaves the global lookup unchanged.
Q. How do you know this works?

A. I implemented it, applied it to every module in library, and ran the
test suite without exception.
Q. Are you the first person to recognize that most global lookups are for
values that never change?

A. No, this has long been known. Skip provides an especially eloquent
explanation in [1]_.
Q. Why not make this a public module so that users can optimize builtins
or builtins and globals in their own code?

A. The ASPN recipe [2]_ is provided for this purpose. It takes a modicum
of skill to use correctly and not inadvertently tranform a non-constant.
Q. What if I have an overpowering urge to import a library module,
write a new shadowing builtin, and insert it into the module's namespace?

A. Your problems are less acute than with a module written in C
because you can always comment out the automatic binding.
Q. More realistically, what if I want to replace the builtins module and
supply my own implementations?

A. Either do this before importing a module, or just reload the module,
or disable _bind.py.
Q. How susceptible is this module to changes in Python's byte coding?

A. It imports opcode.py to protect against renumbering. Also, it uses
LOAD_CONST and LOAD_GLOBAL which are fundamental and have been around
forever. That notwithstanding, the coding scheme could change
dramatically and this implementation would have to change along with
modules like `dis` which also rely on the current coding scheme.

Sample Implementation
=====================

Here is a sample implementation for _bind.py::

from types import ClassType, FunctionType
from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG
LOAD_GLOBAL, LOAD_CONST = opmap['LOAD_GLOBAL'], opmap['LOAD_CONST']

__all__ = ['bind_builtins', 'bind_builtins_and_globals']

def bind(f, builtin_only=True):
""" Return a new function with global references converted to constants.

If builtin_only is True (the default), will convert all builtin
references unless they have been overridden in the function's globals.
If builtin_only is False, will also convert other module variable
references into constants.

"""
import __builtin__
env = vars(__builtin__).copy()
if builtin_only:
stoplist = f.func_globals # do not replace overridden builtins
else:
stoplist = {}
env.update(f.func_globals)

co = f.func_code
newcode = map(ord, co.co_code)
newconsts = list(co.co_consts)
codelen = len(newcode)

i = 0
while i < codelen:
opcode = newcode[i]
if opcode == EXTENDED_ARG:
return f # handle only the simple cases
if opcode == LOAD_GLOBAL:
oparg = newcode[i+1] + (newcode[i+2] << 8)
name = co.co_names[oparg]
if name in env and name not in stoplist:
value = env[name]
try:
pos = newconsts.index(value)
except ValueError:
pos = len(newconsts)
newconsts.append(value)
newcode[i] = LOAD_CONST
newcode[i+1] = pos & 0xFF
newcode[i+2] = pos >> 8
i += 1
if opcode >= HAVE_ARGUMENT:
i += 2

codestr = ''.join(map(chr, newcode))
if codestr == co.co_code:
return f # no changes were found
codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
co.co_flags, codestr, tuple(newconsts), co.co_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstlineno, co.co_lnotab, co.co_freevars,
co.co_cellvars)
return type(f)(codeobj, f.func_globals, f.func_name, f.func_defaults,
f.func_closure)

def bind_all(module_or_class, builtin_only=True):
"""Recursively apply bind() to functions in a module or class.

Use as the last line of the module (after everything is defined, but
before test code or lines defining shorthand references to functions.

If builtin_only is True (the default), will convert all builtin
references unless they have been overridden in the function's globals.

If builtin_only is False, will also convert other module variable
references into constants. This should only be used in modules
where the values of the module variables are truly constant.

"""
for k, v in vars(module_or_class).items():
if type(v) is FunctionType:
newv = bind(v)
setattr(module_or_class, k, newv)
elif type(v) in (type, ClassType):
bind_all(v, builtin_only)

import sys
bind_all(sys.modules[__name__], False) # Binder, bind thyself!

# Self-documenting API (eliminates the builtin_only switch)
bind_builtins = bind_all
bind_builtins_and_globals = lambda m: bind_all(m, False)
The final code should have some minor embellishments:

* Automatic detection of a non-CPython environment that does not have
bytecodes. In that situation, the bind functions would simply
return the original function unchanged. This assures that the two
line additions to library modules do not impact other implementations.

* Add a few lines to handle reading and writing of EXTENDED_ARG. While
this almost never arises in practice, it is best to have the code as
adaptable as possible.
References
==========

... [1] Optimizing Global Variable/Attribute Access
http://www.python.org/peps/pep-0266.html

... [2] ASPN Recipe for a non-private implementation
http://aspn.activestate.com/ASPN/Coo.../Recipe/277940

Copyright
=========

This document has been placed in the public domain.

...
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
Jul 18 '05 #1
8 1847

"Raymond Hettinger" <py****@rcn.com> wrote in message
news:5d**************************@posting.google.c om...
Comments are invited on the following proposed PEP.
Raymond Hettinger


It certainly looks interesting. What's the possibility
of also eliminating the extra function call on global
functions like len() at the same time?

John Roth
Jul 18 '05 #2
Raymond Hettinger wrote:
PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com>
Status: Pre-PEP
Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.



Two things:

1. The new _bind module needs to have a more specific (and
appropriate) name than _bind. The function in the module could
still be named bind, but the module itself should be named
something like

_optimize_globals_as_consts_very_unsafe

(Ok, but you get the idea.)

2. The module being optimized should have a special attribute, say
__variable__, that identifies any globals that should not be
optimized in this way:

__variable__ = ['global_that_might_change']
Other than that, +1. I'm definitely in favor of optimization hacks
whenever it doesn't intrude too much on the code. It would be nice if
it were safer.
--
CARL BANKS http://www.aerojockey.com/software
"If you believe in yourself, drink your school, stay on drugs, and
don't do milk, you can get work."
-- Parody of Mr. T from a Robert Smigel Cartoon
Jul 18 '05 #3
rh> Q. Why not make this a public module so that users can optimize
rh> builtins or builtins and globals in their own code?

rh> A. The ASPN recipe [2]_ is provided for this purpose. It takes a
rh> modicum of skill to use correctly and not inadvertently tranform
rh> a non-constant.

I thought one of the basic tenets of Python was "we're all adults here".
Programmers have unfettered access to the new module with which they can
wreak havoc. I don't see this proposal as somehow any more dangerous.

Skip
Jul 18 '05 #4
Raymond Hettinger wrote:
Comments are invited on the following proposed PEP.
Raymond Hettinger

-------------------------------------------------------

PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com>
Status: Pre-PEP
Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.


Is it really a good thing to do it manually?
Shouldn't be done automatically by a simple analyser?

--
Yermat

Jul 18 '05 #5
Yermat wrote:
Raymond Hettinger wrote:
PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <py****@rcn.com>

Abstract
========

This proposal is to add a private function for treating builtin
references as constants and to apply that function throughout
the standard library.


Is it really a good thing to do it manually?
Shouldn't be done automatically by a simple analyser?


Not requiring that it be done manually, or at least not providing
a simple means of disabling it, would prevent some kinds of
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.

-Peter
Jul 18 '05 #6
On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <pe***@engcorp.com>
wrote:

[snip]
....
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.


Isn't this the use case for AOP you were looking for?
--dang
Jul 18 '05 #7
Daniel 'Dang' Griffith wrote:
On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <pe***@engcorp.com>
wrote:

[snip]
...
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.

Isn't this the use case for AOP you were looking for?
--dang


As you might imagine from my previous postings, I have no idea.
I don't understand well enough what AOP really is to say,
obviously (given my requests for compelling use cases so I can
understand why I would even want to investigate it).

Can you describe how AOP would solve this problem? As I
understand it so far, it has nothing to do with mock objects...

-Peter
Jul 18 '05 #8
On Tue, 20 Apr 2004 10:33:36 -0400, Peter Hansen <pe***@engcorp.com>
wrote:
Daniel 'Dang' Griffith wrote:
On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <pe***@engcorp.com>
wrote:
...
automated testing. The most easily demonstrated one is where
the builtin open/file calls are replaced with a simulated ("mock")
file system which applies to *all* modules, not just a single
module under test. This functionality is critical to some people
using Python with the Test-Driven Development approach to software.


Isn't this the use case for AOP you were looking for?
--dang


As you might imagine from my previous postings, I have no idea.
I don't understand well enough what AOP really is to say,
obviously (given my requests for compelling use cases so I can
understand why I would even want to investigate it).

Can you describe how AOP would solve this problem? As I
understand it so far, it has nothing to do with mock objects...


Probably not.
I'm not an "AOP person", but from what I understand, you could use
an AOP-enabled language to replace the real file system (or whatever)
objects with the mock objects. This is roughly analagous to the
standard logging or auditing examples, wherein a program is doing what
it needs to do, meanwhile a cross-cutting aspect intercepts certain
operations and performing additional, or different, operations.
--dang
Jul 18 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

59
by: Raymond Hettinger | last post by:
Please comment on the new PEP for reverse iteration methods. Basically, the idea looks like this: for i in xrange(10).iter_backwards(): # 9,8,7,6,5,4,3,2,1,0 <do something with i> The...
36
by: Gonçalo Rodrigues | last post by:
I've lost the original thread on my news reader so I'm opening a new one. First, i like the name ireverse, much better then inreverse (eek!). Just a simple comment/question: - Why a builtin...
0
by: Raymond Hettinger | last post by:
Comments are invited on a new pep: http://www.python.org/peps/pep-0329.html Based on feedback received so far, there were several changes to the original draft: * The module is public
38
by: Kevin Smith | last post by:
For what it's worth, I wrote the original PEP 318. I probably wasn't qualified, but I just wanted a nice simple way to declare class methods without having to repeat the function name. After...
108
by: Bryan Olson | last post by:
The Python slice type has one method 'indices', and reportedly: This method takes a single integer argument /length/ and computes information about the extended slice that the slice object would...
77
by: Ben Finney | last post by:
Howdy all, PEP 354: Enumerations in Python has been accepted as a draft PEP. The current version can be viewed online: <URL:http://www.python.org/peps/pep-0354.html> Here is the...
4
by: Tony Lownds | last post by:
(Note: PEPs in the 3xxx number range are intended for Python 3000) PEP: 3107 Title: Function Annotations Version: $Revision: 53169 $ Last-Modified: $Date: 2006-12-27 20:59:16 -0800 (Wed, 27 Dec...
4
by: dustin | last post by:
I've been hacking away on this PEP for a while, and there has been some related discussion on python-dev that went into the PEP: ...
399
by: =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= | last post by:
PEP 1 specifies that PEP authors need to collect feedback from the community. As the author of PEP 3131, I'd like to encourage comments to the PEP included below, either here (comp.lang.python), or...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.