469,327 Members | 1,333 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,327 developers. It's quick & easy.

rotor replacement

I see rotor was removed for 2.4 and the docs say use an AES module
provided separately... Is there a standard module that works alike or
an AES module that works alike but with better encryption?

cheers,
reed

Jul 18 '05
92 5821
phr
"Martin v. Löwis" <ma****@v.loewis.de> writes:
There isn't. The interface might be beautifully designed, and you might
claim it is, and I would *still* require that the module gets field
testing before being incorporated into Python.
Yes, of course, that's completely reasonable.
If other people start attesting that the module is beatifully
designed, and should be included in the Python core - *then* it is
worth looking into inclusion.
You said something more than that. You said it's not worth even
thinking about inclusion until the finished module is actually in
distribution outside the core. That it's not appropriate for the
distro maintainers to look at the spec and the reference (pure Python)
implementatation and say "yes, we want this, go write the C version
and we'll include it after it's had some testing". I think that's
silly. (Anyway I don't think "beautifully designed" needs to be a
requirement, and I don't claim to have done a beautiful design. "Gets
the job done straightforwardly" is all that's needed and is all that
I've aimed for).

When the inestimable Alexbot went to O'Reilly to pitch "Python in a
Nutshell", I'm sure they didn't tell him "go write the whole book from
cover to cover, circulate it for at least a year and get it thoroughly
reviewed, and if enough people recommend it, then and only then will
we begin to think about distributing it ourselves". More normally,
he'd have given them a proposal with an outline and maybe a sample
chapter and they used that to decide to give him a go-ahead. Usually
at that point, they'd even also pay him some money in advance to do
the rest of the work with, and promise to pay him a cancellation fee
if they changed their mind about the book.

I certainly never asked for anything like that from the Python guys
about the crypto module. I did ask for, and receive, a "this looks
good, we want it". I later heard that they backed off because of
legal concerns, so I put the task aside.

Similarly, look at how the whole PEP process works. There are lots of
times when a PEP has been accepted before any working code is
distributed.

To take your view to an extreme, no project should even have a task
list of desired features that people are invited to implement.
There's been numerous times when I've written something for the GNU
project because it's been on a list like that, and it's been accepted
after checking that it does what it's supposed to.
I'm happy to have that kind of testing (and I requested it), given
that the goal is inclusion in the core, and the core developers have
told me (as they did) that the proposal looks good and they'd like to
have the module, so I can reasonably expect it to go into the core if
it meets its technical expectations.


Not if I have a say in it. *Any* new module should see out-of-the-core
distribution first (unless there is BDFL pronouncement to include it,
of course).

This really is a matter of development process, not of technical
quality.


That's bizarre and abnormal as a development process. What kind of
development process in industry doesn't decide whether to include a
feature, until after the feature is completely implemented at a
production scale? They figure out what the features should be, and
write them into a specification. The specification defines what
features will be included. Then they implement the features. (I'm
talking specificially about straightforward, well-defined features,
not blue-sky or experimental concepts).

You see, they decide in advance what features they're going to
include, because development consumes resources and they don't want to
burn resources to develop something unless they're pretty sure they're
going to use the result once it's done. (Of course it doesn't always
work out that way. Everyone has worked on projects that have gotten
cancelled before delivery. However that by definition means that
something has gone wrong. No engineer likes it when that happens.)

You seem to have the attitude that since volunteer development effort
doesn't consume actual PSF funds, the volunteer effort is worth
nothing and can be expended arbitrarily. The volunteers may not feel
that way.
If the developers instead say (as they seemed to somewhat later) that
because of legal/political concerns, there's no way the module can
possibly go into the core no matter how good it is technically, then
my motivation for writing the module dries up quite a bit.


I personally would not say that, although I can imagine that some people
do say that, and I would also defend an inclusion, and push compliance
the BXA requirements so we can legally export Python out of the U.S.A.


The BXA notification on the US side is pretty easy to handle, as you
know. If you look at those Python-dev messages, the concern was about
importing Python-with-crypto into countries like Holland, not about
exporting it from the USA. Those concerns are not necessarily
something to sneeze at. Salman Rushdie had to go into hiding for
years, after publishing a book in England whose contents were illegal
in Iran. So, I can't get too upset with the Python maintainers for
deciding to stay away from such political messes, even though I don't
feel such strong concern myself.
In either case, the user would best use the pre-compiled binary that
somebody else provided for the platform.
Who's the "somebody else"? The module author might also not have had
a compiler for that platform. Are you suggesting that he maintain his
own distribution of multiple binaries, duplicating the machinery of
the Python distro for managing multi-platform builds? Do you do that
with your own modules, and still say that it's easy? If you don't,
then how does the end user know where to get the module for his
platform?
No, I'm merely repeating myself, and rephrasing each time.
I have to, because apparently you don't see what my requirement is.
Unless your requirement is different than what you say it is, I do see
what it is, and I'm saying it's better to do what normal projects do
and what Python has done in the past. That is, it's perfectly ok to
decide to do something and then do it, rather than insisting,
bizarrely, that it always be the other way around.
The original source code of the SHA-1 implementation is the NIST code
(Gutmann, then Hollerbach), so I guess that had hundreds of users before
the module was contributed to Python.
Not terribly relevant, the NIST code was not a Python module.
The module itself (including the API) was written by Greg Stein and
Andrew Kuchling. I believe (without being able to verify) that they
distributed this module for quite some time, before contributing it
to Python. We would have to ask them how many users they had until
they felt confident to contribute the code.


I think that question isn't the right one. We need to ask how many
users the sha module was required to have, before Greg and Andrew
could have reasonable confidence that the sha module would go into the
core once it was tested enough and shown to be reliable. I suspect
they were able to have that confidence long before testing was
complete, and maybe before implementation even started. That is, they
were likely able to decide that the core obviously needed a sha
module, before setting off to write and test one for that purpose.

If Andrew is reading this, maybe he can comment.
Jul 18 '05 #51
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
jj*@pobox.com (John J. Lee) writes:
Building larger ones seems to
have complexity exponential in the number of bits, which is not too
Why?


The way I understand it, that 7-qubit computer was based on embedding
the qubits on atoms in a large molecule, then running the computation


Oh, you mean that particular kind, OK. Doesn't apply to QC in
general.

It's not even known in theory whether quantum computing is
possible on a significant scale.


Discuss. <wink>


The problem is maintaining enough coherence through the whole
calculation that the results aren't turned into garbage. In any
physically realizeable experiment, a certain amount of decoherence
will creep in at every step. So you need to add additional qubits for
error correction, but then those qubits complicate the calculation and
add more decoherence, so you need even more error correcting qubits.


Yes, that's much more interesting, dunno what the current state of
play is.

[...] I'm not any kind of expert in this stuff but have had some
conversations with people who are into it, and the above is what they
told me, as of a few years ago. I probably have it all somewhat garbled.


Me too :-)
John
Jul 18 '05 #52
ph*@localhost.localdomain wrote:
That it's not appropriate for the
distro maintainers to look at the spec and the reference (pure Python)
implementatation and say "yes, we want this, go write the C version
and we'll include it after it's had some testing".
I know that I'm not going to give a blanket promise to include some
code in the future. I can give blanket promises to *review* such
code. Actually, assuming I find the time, I will review *any*
code that is contributed.
When the inestimable Alexbot went to O'Reilly to pitch "Python in a
Nutshell", I'm sure they didn't tell him "go write the whole book from
cover to cover, circulate it for at least a year and get it thoroughly
reviewed, and if enough people recommend it, then and only then will
we begin to think about distributing it ourselves".
Sure. That is because O'Reilly is willing to take a financial risk
of failure of a book product. I'm not willing to take similar risks
for Python code (risk of having to redesign the interface, fix bugs
in the implementation, or have the submitter run away after the
contribution).
Similarly, look at how the whole PEP process works. There are lots of
times when a PEP has been accepted before any working code is
distributed.
Indeed. For new language features, it is more difficult to try them out
in advance than for new library API. As a result, flaws of the new
feature are often only fully understood years after the new feature
first gets released. For an example, consider the buffer interface.
To take your view to an extreme, no project should even have a task
list of desired features that people are invited to implement.
Taking it to the extreme misses the point. I'm asking for field testing
for new modules - not for all changes.
That's bizarre and abnormal as a development process. What kind of
development process in industry doesn't decide whether to include a
feature, until after the feature is completely implemented at a
production scale?
Any high-quality standardization process. Standards in IETF and OMG
are only accepted after implementations have been available.
You seem to have the attitude that since volunteer development effort
doesn't consume actual PSF funds, the volunteer effort is worth
nothing and can be expended arbitrarily. The volunteers may not feel
that way.
The volunteers are free to work on whatever they please. If you chose
not to write an AES module - that's fine with me. Still, people do
contribute to Python, and they do so without asking for permission
first. Typically, they develop the code because it solves their own
needs - then it doesn't really matter whether it also solves the
needs of others.
In either case, the user would best use the pre-compiled binary that
somebody else provided for the platform.

Who's the "somebody else"?


Some user. I find that users contribute binaries for all kinds of
platforms for the code I publish. This is how open source works.
Do you do that
with your own modules, and still say that it's easy?
I publish or link to binaries of my code that others have created,
and find no problems in doing so.
Unless your requirement is different than what you say it is, I do see
what it is, and I'm saying it's better to do what normal projects do
and what Python has done in the past. That is, it's perfectly ok to
decide to do something and then do it, rather than insisting,
bizarrely, that it always be the other way around.
No, it's three steps
1. decide that you want to do it
2. do it
3. decide whether you are pleased with the result, and only
use it if you are

IOW, there should not be a blanket guarantee to use it after
step 1.
I think that question isn't the right one. We need to ask how many
users the sha module was required to have, before Greg and Andrew
could have reasonable confidence that the sha module would go into the
core once it was tested enough and shown to be reliable.
They did not have *any* guarantee until they asked. I guess when they
asked it was accepted immediately.
I suspect
they were able to have that confidence long before testing was
complete, and maybe before implementation even started.


I'm pretty certain that you are wrong with that assumption.
Again, we would have to ask - but I would not be surprised if
AMK started implementing the module without even *considering*
a later inclusion in the Python core at that time. He has done
so on many occasions (include PyXML, which I inherited from
him).

Regards,
Martin
Jul 18 '05 #53
phr
"Martin v. Löwis" <ma****@v.loewis.de> writes:
That it's not appropriate for the
distro maintainers to look at the spec and the reference (pure Python)
implementatation and say "yes, we want this, go write the C version
and we'll include it after it's had some testing".
I know that I'm not going to give a blanket promise to include some
code in the future. I can give blanket promises to *review* such
code. Actually, assuming I find the time, I will review *any*
code that is contributed.


I don't see why you can't make up your mind enough to issue simple
statements like "the Python lib should have a module that does
so-and-so, and it should meet such-and-such requirements, so if
someone submits one that meets the requirements and passes code review
and testing and doesn't have unexpected issues or otherwise fail to
meet reasonable expectations, we'll use it".
Sure. That is because O'Reilly is willing to take a financial risk
of failure of a book product. I'm not willing to take similar risks
for Python code (risk of having to redesign the interface,
Again, we're talking about straightforward modules whose basic
interface needs are obvious. And interfaces in stdlib do get extended
from version to version all the time, if users turn out to need
additional features beyond the obvious basics.
fix bugs in the implementation,
Obviously there must be testing and review before inclusion.
Acceptance is contingent on the module passing tests and review.
or have the submitter run away after the contribution).
There is no way to know in advance whether that's going to happen. A
lot of work on the ANSI X9 crypto standards came to a screeching halt
a few years ago when one of the more prolific contributors tripped
over his vacuum cleaner cord, fell down the stairs, and was killed.
So if you have to be absolutely sure that the submitter will always be
around, you can never accept anything. I think you mostly have to go
by how maintainable the code looks and how much maintainance you think
it will actually need and how many people you think are around who can
take care of it when needed. And I do believe that experienced
programmers are capable of making reasonable judgements about those
questions, so they should not refuse to ever make such judgements.
Similarly, look at how the whole PEP process works. There are lots of
times when a PEP has been accepted before any working code is
distributed.


Indeed. For new language features, it is more difficult to try them out
in advance than for new library API.


I don't see why that should be. Nothing stops anyone from
implementing and testing a language feature before standardizing it.
It will get even easier with PyPy, which is perhaps a reason to ban
further language changes until PyPy is deployed.
Taking it to the extreme misses the point. I'm asking for field
testing for new modules - not for all changes.
"Field testing" to most people means testing that a specific
implementation is reliable enough for inclusion. It is the final step
in a normal process of declaring a feature ready for deployment, not
the initial step:

1) think about whether you want the feature
2) decide you want it
3) implement
4) field test (this naturally recognizes the possibility of
reversing step 2, if something unexpectedly goes wrong in testing
that's not easily repaired, but step 2 declares the basic
intention for what should happen after a successful test).
5) deploy

You wanted much more than for step 4 to always happen before step 5,
which is reasonable. You claimed step 4 should always happen before
step 1, which is silly.
That's bizarre and abnormal as a development process. What kind of
development process in industry doesn't decide whether to include a
feature, until after the feature is completely implemented at a
production scale?


Any high-quality standardization process. Standards in IETF and OMG
are only accepted after implementations have been available.


I don't know what OMG is, but there is no IETF requirement that any
implementations be available in any particular language. There are
also plenty of instances where the IETF decides that it wants
something to be standardized (e.g. IPSEC) so it invites a working
group to develop a specification. The WG members then spend a lot of
time in meetings and discussions reaching a consensus on what the spec
should say. They are willing to spend that time because the IETF has
already given them reasonable expectation that their end result will
actually be used. The IETF doesn't say "go develop a complete
standard and implementation and put it in the field for a year before
we [the IETF] will even think about whether we want to standardize
it". They are capable of announcing in advance that they want to
standardize something. I don't see why the Python folks have to be
incapable of ever doing the same.

For the module we're discussing, there is already a published working
implementation, written in Python. It implements every feature of the
API and has been integrated into various applications/demos and tested
with them. And, there was reasonable consensus on clpy and on the
python-crypto list that the API did the right things. That is all
that the IETF normally requires ("rough consensus and working code").
The Python implementation is unsuitable for the stdlib because it's
too slow to use in serious production apps and so it needs to be
rewritten in C, not because anything is missing or untested about it.
(There's also an unpublished hybrid Python/C implementation that's
tolerably fast, but unsuitable for contribution because it depends on
external packages that can't go in the Python distro). But the IETF
would not care about that. All they care is that useable
implementations exist.

Finally, the IETF generally only specifies protocols and doesn't care
about either the implementation specifics or the API (they require
implementations to exist only in order to prove that the protocol is
usable). The protocols here would be the FIPS operating modes (ECB,
CBC, CFB, etc). Those are already standardized as FIPS and have test
vectors published by NIST. The Python implementation computes all the
published test vectors correctly and interoperates with various
non-Python programs that use those same modes. So again, there is
already enough available to satisfy an IETF-like standardization
process. Your IETF example does not support your stance.
The volunteers are free to work on whatever they please. If you chose
not to write an AES module - that's fine with me.
However, the result of my not writing an AES module is that Python
doesn't have an AES module. Because as far as I can tell, while
numerous people want to use such a module, nobody except me has
expressed willingness to write and contribute one. If someone else
did it, I would be overjoyed, and I'd happily use whatever they wrote,
if it was useable at all, dropping any plans to write my own. (And
frankly, the only feature such a module absolutely needs to provide is
simple ECB mode block encryption and decryption. The other modes are
helpful for good performance and convenience but can be done as pure
Python functions that call the ECB mode primitive, with tolerable
speed for lots of useful apps. It's just the block cipher primitive
that's painfully slow in Python).
Still, people do contribute to Python, and they do so without asking
for permission first.
They have not done that for an AES module. That is a fact on the
ground. I don't have any burning desire to be the author of a Python
AES module. I just think Python should have one that I and other app
writers can rely on, so I've been willing to volunteer to write and
contribute it, since I'm qualified and nobody else has stepped
forward. But there appears to currently be a policy in force saying
"there will be no crypto module in the Python stdlib regardless of its
technical quality". Under that policy, there's no point in my writing
one to contribute it.
Typically, they develop the code because it solves their own needs -
then it doesn't really matter whether it also solves the needs of others.
In fact that's been done with AES and DES numerous times, and the
resulting modules do meet my needs for my own end-use (I'm currently
using a C library wrapped with SWIG). What's missing is a module in
the stdlib that lets me ship pure-Python crypto apps that other end
users can run without installing C code. And, as we discussed
already, the modules people have written for their own needs (at least
so far) aren't really technically suitable for the stdlib, because a
good general purpose stdlib module has somewhat different
characteristics than a module written to support a specific
application, which is what people write to solve their own needs.

Note that the same thing is true for most other types of library
modules besides crypto (i.e. a general purpose module suitable for the
stdlib tends to be different from what someone writes to solve their
own needs). So either most stdlib modules were either not written
purely to solve the author's needs and then submitted as an
afterthought as you describe, or else the modules in the stdlib
typically don't have good general-purpose designs. (In fact the
latter is often true, which suggests that the practice of contributing
modules originally written only to solve an individual need often
results in lousy stdlib modules, and a policy change towards
encouraging designing intentionally for the stdlib is probably a good
thing. To paraphrase RMS, a good stdlib needs to come from a vision
and a plan, not just from scratching a bunch of different itches).
In either case, the user would best use the pre-compiled binary that
somebody else provided for the platform.

Who's the "somebody else"?


Some user. I find that users contribute binaries for all kinds of
platforms for the code I publish. This is how open source works.
Do you do that
with your own modules, and still say that it's easy?


I publish or link to binaries of my code that others have created,
and find no problems in doing so.


And you seriously believe that the result is as painless for end users
as having a module in the stdlib that apps can just import and call
without the end-user having to first locate, download and install a
3rd party module? Do you think the Python philosophy of "batteries
included" really is a) meaningless, b) worthwhile in the past but now
obsolete, c) a mistake to begin with, d) sort of worthwhile but not
really important, e) something else? That's not a rhetorical
question, I'm really having a hard time figuring out what you think
"batteries included" is supposed to mean. I personally consider it
meaningful and extremely important, or else I'd be using Scheme
instead of Python.
No, it's three steps
1. decide that you want to do it
2. do it
3. decide whether you are pleased with the result, and only
use it if you are

IOW, there should not be a blanket guarantee to use it after step 1.
But, it's completely normal to say before step 1 that "if the result
of step 2 does so-and-so, then I'll be pleased in step 3", stating
some clear requirements and sticking to them. People do that in real
life all the time, including in standardization processes, and often
with ironclad guarantees (called "contracts").

It's pathetic if the Python maintainers are so indecisive as to be
incapable to ever saying (even -without- making ironclad guarantees),
that "yeah, we really ought to have an XYZ module and we'll use one if
someone submits it and it does so-and-so", giving an informal
assurance to someone thinking of doing an implementation that they're
probably not wasting their time. That's often what allows step 1 to
happen.
I think that question isn't the right one. We need to ask how
many users the sha module was required to have, before Greg and
Andrew could have reasonable confidence that the sha module would
go into the core once it was tested enough and shown to be
reliable.


They did not have *any* guarantee until they asked. I guess when they
asked it was accepted immediately.


And who do you think they asked, hmm? When I asked Guido about
submitting a crypto module, he told me that he defers all technical
crypto issues to Andrew. So I think Andrew had a reasonable
expectation of what would happen when he submitted it. You and
Frederik seem to think there's something inappropriate or
self-inflated about wanting that expectation before committing to do a
pile of work that's primarily for other people's benefit. I think
your stated attitude is completely bizarre, that you can't really
believe anything so silly, so you're really just acting bureaucratic,
looking for excuses to say no instead of yes to worthwhile proposals.
Again, we would have to ask - but I would not be surprised if
AMK started implementing the [sha] module without even *considering*
a later inclusion in the Python core at that time. He has done
so on many occasions (include PyXML, which I inherited from him).


I would be surprised. That need for an sha module was completely
obvious, what it needed to do was completely obvious, and the
requirements and implementation have nothing like the subtlety of PyXML.
Jul 18 '05 #54

phr> I don't see why you can't make up your mind enough to issue simple
phr> statements like "the Python lib should have a module that does
phr> so-and-so, and it should meet such-and-such requirements, so if
phr> someone submits one that meets the requirements and passes code
phr> review and testing and doesn't have unexpected issues or otherwise
phr> fail to meet reasonable expectations, we'll use it".

Because good requirements specification is difficult and testing improves
the breed. Better to have the major API changes and bugs taken care of, and
to have its popularity demonstrated *before* it gets into the Python
distribution. The best way to do something isn't always obvious at the
outset. If multiple solutions exist in the community, everyone benefits.
The best survive. The worst don't. If one implementation is prematurely
chosen for inclusion in the Python distribution, it stifles the vetting
process. Finally, what if, saints be preserved, your whizbang new module is
included in the core distribution and it's just not as popular as was first
thought? It just winds up as a weight around the maintainers' necks. Once
included in the distribution it's difficult to remove. Even rexec and
Bastion, which are both known to be inadequate from a security standpoint,
are still in the distribution.

So, feel free to implement whatever it is you propose. Register it with
PyPI, announce it on comp.lang.python.announce, etc. Support it for awhile.
Run it through a couple revisions to fix API warts and bugs. When it's the
category king and there is substantial community support for inclusion, it
will be considered.

Python is popular in part because of its fairly conservative development
strategy. That goes for the libraries as well as the language itself.

Skip
Jul 18 '05 #55
phr
Skip Montanaro <sk**@pobox.com> writes:
Because good requirements specification is difficult and testing improves
the breed. Better to have the major API changes and bugs taken care of, and
to have its popularity demonstrated *before* it gets into the Python
distribution. The best way to do something isn't always obvious at the
outset.
That is sometimes true, but not always. Sometimes the best way is
obvious.
If multiple solutions exist in the community, everyone benefits.
The best survive. The worst don't.
What's needed here "good enough", not necessarily "the best
conceivable", a level of perfection which is perhaps unattainable.
If one implementation is prematurely chosen for inclusion in the
Python distribution, it stifles the vetting process.
There have been exactly zero alternative implementations proposed.
Finally, what if, saints be preserved, your whizbang new module is
It is not a whizbang module. It is a stripped-down, basic
implementation of a well-accepted set of standards that are being used
in thousands of other applications in other languages.
included in the core distribution and it's just not as popular as
was first thought?
There is demand for it. Look at how this thread started: some crypto
user noticed that rotor was gone and wanted to know what to do instead.

The issue of whether there's enough desire for a crypto module to
warrant including one in the stdlib was decided a long time ago. The
proof of that somebody decided to accept the rotor module into the
distro.

That decision, that enough users want crypto to make having a stdlib
module worthwhile, has never been reversed as far as I know. Rather,
the specific module (rotor) was withdrawn because it turned out not to
do the job on technical grounds. And the decision not to replace it
was based on new legal and political concerns that outweighed the
technical side, not on thinking that there wouldn't be enough users.
It just winds up as a weight around the maintainers' necks. Once
included in the distribution it's difficult to remove.
The rotor module is gone (and good riddance). That's how this thread
started, remember? It shows that bogus modules can be removed.

Have you ever used a crypto library in a serious way? Which ones have
you used? The python-crypto folks (including myself) have used quite
a few of them, and after a while one realizes that the usable ones all
do pretty much the same stuff. The one we're talking about does about
the same stuff that the rest of them do. It does the same stuff
listed in PEP 272, although it's organized a little bit differently in
order to make supplying multiple cipher primitives more convenient,
which the application that drove PEP 272 apparently didn't need to do.
Andrew (the author of PEP 272) has said that the different
organization is fine. The different organization is similar to what's
done internally in OpenSSL, in Peter Gutmann's cryptlib, in GnuPG, and
various other successful programs. It's not radical or experimental.
It's old hat and low risk. It provides functionality that's missing
from the stdlib.
Even rexec and Bastion, which are both known to be inadequate from a
security standpoint, are still in the distribution.
They should be removed pronto, just as rotor was removed. Some other
ones should be removed too, like SmartCookie.
So, feel free to implement whatever it is you propose.
I don't need your permission to do that. What I'm looking for is an
indication that it's worth my while, before I spend the effort.
Register it with PyPI, announce it on comp.lang.python.announce, etc.
I don't know how to do that, but the folks who care about it are
mostly on the python-crypto mailing list, and the module has been
discussed at length there and has been well-received.
Support it for awhile. Run it through a couple revisions to fix API
warts and bugs.
I already did that with the Python implementation, and did in fact
revise the API somewhat based on reviewer feedback. The C
implementation would have the exact same API that the reviewers seem
to be happy with.
When it's the category king and there is substantial community
support for inclusion,
It's already the category king, because there are precisely zero other
entrants in the category. If someone else wants to do one and can get
it into the stdlib, I will gladly use theirs and stop worrying about
mine.
it will be considered.
Not according to how I currently interpret Guido's posts to python-dev
about it. I read those as saying that no crypto module will be
considered for inclusion whether or not it's the category king,
because any such module might conflict with crypto regulations in some
countries. No matter how much community support or technical
perfection it has, it's just won't be included, period. So tell me
again what to do after writing and releasing a C module. There's just
no reason to write one, if the module can't go into the stdlib. The
stdlib has some special requirements but there are existing crypto
modules that I'm using and that work just fine outside of the stdlib,
even though they don't meet the stdlib's special needs. The only
reason to write a special module just to meet the stdlib's special
needs is so that the stdlib can use it.
Python is popular in part because of its fairly conservative development
strategy. That goes for the libraries as well as the language itself.


Tell me again how rotor got into the distribution.
Jul 18 '05 #56
On Thu, 27 Jan 2005 04:04:38 +0000, phr wrote:
Skip Montanaro <sk**@pobox.com> writes:
Because good requirements specification is difficult and testing improves
the breed. Better to have the major API changes and bugs taken care of, and
to have its popularity demonstrated *before* it gets into the Python
distribution. The best way to do something isn't always obvious at the
outset.
That is sometimes true, but not always. Sometimes the best way is
obvious.


With all due respect, you aren't arguing very well. Your posts are, like
this quote, more reactionary than thought-out.

You need to stop arguing for the sake of argument, read more carefully,
and think more about your global goals in this conversation than your
local ones. I'm 99% certain that you've already gotten all the answers
you're going to get and all the answers you need in this thread, but
you're so busy arguing the local points like this, you haven't noticed.

(Either that, or it's the "you haven't answered me because you haven't
given the exact answer I want" bit, which I find quite tiresome; there's
another list where there are these three teenagers that don't even realize
they've defined "answer" that way and keep bitching that they aren't being
"answered" when the "answer" is, basically, "it's nice that you feel that
way." But I digress.)
I don't need your permission to do that. What I'm looking for is an
indication that it's worth my while, before I spend the effort.


The policy has been laid out, multiple times, by multiple people now. The
answer is, you are not going to get any such indication that will satisfy
you. Note that I am not a Python contributor of any kind. Also note that I
figured this out about two days ago. You can wheedle these guys all you
want, but they are too experienced for you to extract a promise from them.
Jul 18 '05 #57
phr
Jeremy Bowers <je**@jerf.org> writes:
The policy has been laid out, multiple times, by multiple people now. The
answer is, you are not going to get any such indication that will satisfy
you.
Actually I already got an indication that satisfied me, from Guido and
Andrew, although it was later withdrawn for nontechnical reasons (i.e.
legal restrictions) that I don't feel entitled to argue against very
much. I do, however, believe that what Frederik and Martin are saying
is bogus on technical grounds, and furthermore, they're just being
busybodies, since they've had no involvement at all in what's happened
already with that module, and their picture of the Python development
process doesn't have much resemblance to how the current Python distro
got to be the way it is.
Note that I am not a Python contributor of any kind. Also note that I
figured this out about two days ago. You can wheedle these guys all you
want,
I have not asked them for anything, since they are not in a position
to give it.
but they are too experienced for you to extract a promise from them.


Actually, one of them already gave a promise that he couldn't keep
even if he wanted to. He said that if I wrote a crypto module and got
it tested enough, it would be considered for inclusion (i.e. based on
its technical merits). But, he doesn't get to decide that. Current
policy per per Guido seems to be that because of the legal stuff,
there will be no crypto module in the stdlib regardless of its merits.
Jul 18 '05 #58
>> Finally, what if, saints be preserved, your whizbang new module is

phr> It is not a whizbang module. It is a stripped-down, basic
phr> implementation of a well-accepted set of standards that are being
phr> used in thousands of other applications in other languages.

Then there should be a library already out there already. All you should
need to do is wrap it (manually, with SWIG, whatever).
included in the core distribution and it's just not as popular as was
first thought?
phr> There is demand for it. Look at how this thread started: some
phr> crypto user noticed that rotor was gone and wanted to know what to
phr> do instead.

Yes, and putting rotor back would be the wrong thing to do.

phr> The issue of whether there's enough desire for a crypto module to
phr> warrant including one in the stdlib was decided a long time ago.
phr> The proof of that somebody decided to accept the rotor module into
phr> the distro.

No, rotor was added in Python's early days (July 1992). Times have changed.

As long as we are discussing cryptography, what's wrong with m2crypto?

http://sandbox.rulemaker.net/ngps/m2/

Why not incorporate it into the standard distribution?

Or, what about Andrew Kuchling's crypto toolkit?

http://www.amk.ca/python/code/crypto.html

I believe both have been around awhile. If crypto-in-the-core is really
what's needed why not see if one of them is ready to go?

phr> The rotor module is gone (and good riddance). That's how this
phr> thread started, remember? It shows that bogus modules can be
phr> removed.

Yeah, but it was there for over 10 years.

phr> Have you ever used a crypto library in a serious way?

Nope, never directly. Don't make this about me. I'm interested in the
Python development process and how you'd like to turn it on its head.
When it's the category king and there is substantial community
support for inclusion,
phr> It's already the category king, because there are precisely zero
phr> other entrants in the category.

See my above references. Note, I don't use crypto at all, yet I was aware
of both of these (no Googling required). My guess would be they are
substantially more mature than your proposed module.

phr> I read those as saying that no crypto module will be considered for
phr> inclusion whether or not it's the category king, because any such
phr> module might conflict with crypto regulations in some countries.

That may be a problem, sure. I'm not sure how the discussion here changes
that. That's just life as we know it.

phr> So tell me again what to do after writing and releasing a C module.
phr> There's just no reason to write one, if the module can't go into
phr> the stdlib.

Why in the heck is inclusion in the standard library a requirement for you
to write this thing? If it's useful to you, write it and get on with your
life.
Python is popular in part because of its fairly conservative
development strategy. That goes for the libraries as well as the
language itself.


phr> Tell me again how rotor got into the distribution.

Okay. It was 1992. Bill Clinton had recently been elected president. It
was pretty much pre-WWW as we know it. Definitely pre-comp.lang.python and
pre-Google (heck, pre-Yahoo, pre-Win98 and pre-Mac OSX as well). Pre-string
methods. Pre-triple-quoted strings. Pre-spammers. Pre-DSL. Pre-lots of
stuff. There was an Emacs python-mode and <wink>s, both thanks to Tim.
People opined about Python's performance, just as they do today. Python's
version number was around 0.9.4, definitel < 1.0. Guido was the only person
with direct repository access. Including something in the distribution was
probably the only convenient way to make new modules available. If nothing
else, the rotor module (along with anything else included in the
distribution back then) may have been a good exercise in and demonstration
of writing extension modules, so it probably served a useful non-crypto
purpose. Python's user community was probably a few hundred people. Guido
likely had no thoughts of world domination with the little language that
could. Times have changed. You seem think there was a PEP process and
distutils and Wikis. I suspect some of the algorithms one might include in
a robust crypto toolkit today weren't even invented in 1992.

So throw away the rotor crutch and put your money where your mouth is.

Skip
Jul 18 '05 #59
<snip>

As long as we are discussing cryptography, what's wrong with m2crypto?

http://sandbox.rulemaker.net/ngps/m2/

Why not incorporate it into the standard distribution?

Or, what about Andrew Kuchling's crypto toolkit?

http://www.amk.ca/python/code/crypto.html


<snip>

Umm, is it just me or did we just discuss the legal issues of that??
Jul 18 '05 #60
>> As long as we are discussing cryptography, what's wrong with m2crypto?
Or, what about Andrew Kuchling's crypto toolkit?


Lucas> Umm, is it just me or did we just discuss the legal issues of
Lucas> that??

You may have. Whether or not there are legal issues with them is of little
relevance to the point I was making. Anything Paul writes would probably
have the same legal entanglements. I was simply pointing out that maybe,
just maybe, there are already suitable candidates from a technical
standpoint and that he doesn't need to write anything.

Skip

Jul 18 '05 #61
phr
Skip Montanaro <sk**@pobox.com> writes:
phr> It is not a whizbang module. It is a stripped-down, basic
phr> implementation of a well-accepted set of standards that are being
phr> used in thousands of other applications in other languages.

Then there should be a library already out there already. All you
should need to do is wrap it (manually, with SWIG, whatever).
I'm currently using something wrapped with SWIG, but my understanding
is that core modules are not supposed to depend on SWIG. So a core
module will likely use some existing primitives wrapped by hand. That
is what I've offered to do, along with providing a somewhat more
Pythonic interface (though still a straightforward one) than directly
wrapping a C library intended for use from C applications.
phr> There is demand for it. Look at how this thread started: some
phr> crypto user noticed that rotor was gone and wanted to know what to
phr> do instead.

Yes, and putting rotor back would be the wrong thing to do.
Correct. The right thing is to replace rotor with something
reasonable that follows standards.
phr> The issue of whether there's enough desire for a crypto module to
phr> warrant including one in the stdlib was decided a long time ago.
phr> The proof of that somebody decided to accept the rotor module into
phr> the distro.

No, rotor was added in Python's early days (July 1992). Times have changed.
I don't see that. There's surely even more demand for crypto now than
there was in 1992.
As long as we are discussing cryptography, what's wrong with m2crypto?

http://sandbox.rulemaker.net/ngps/m2/
It's a good package but it's pretty heavyweight. It depends on both
SWIG and OpenSSL. I think it's still under development--there's an
occasional flurry of messages about it on python-crypto, but I haven't
been following it closely. I'd have a hard time making a case for
accepting it into the core given the difficulty I'm having making the
case for something as simple as a block cipher wrapper.
m2crypto+OpenSSL is at least 100 times as much code as the module I've
proposed. I think the Python lib should someday have its own
pure-Python SSL/TLS implementation sort of like the one Java has. But
if m2crypto went into the lib, I'd use it.
Why not incorporate it into the standard distribution?
I don't have the authority to incorporate anything into the standard
distribution. All I can do is offer to contribute stuff that I write,
and let the distro maintainers decide whether to incorporate it.

I don't have the authority to offer m2crypto, since I'm not the author.
Only the authors can do that. They haven't done so, as far as I know.
Or, what about Andrew Kuchling's crypto toolkit?

http://www.amk.ca/python/code/crypto.html
This is perhaps more suitable than m2crypto but as far as I know,
Andrew hasn't offered to contribute it. Whatever his reasons are, I
have to respect them. I think it has more stuff than a core crypto
module really needs (e.g. numerous semi-obsolete algorithms that
aren't in widespread use so aren't needed for interoperability) but
the extra stuff doesn't really get in the way. If it were in the
core, it would completely fulfill my desires and I would be
transported with ecstacy. But I've never seen any indication that
it's headed there.
I believe both have been around awhile. If crypto-in-the-core is
really what's needed why not see if one of them is ready to go?
I don't think m2crypto is the answer, though maybe I'm wrong. And if
Andrew's package is the answer, he would have submitted it already.
phr> Have you ever used a crypto library in a serious way?

Nope, never directly. Don't make this about me. I'm interested in the
Python development process and how you'd like to turn it on its head.
Either you're the one turning the actual process on its head, or else
it's already on its head and needs to be turned rightside up. Either
way, the existing process has certainly been a total failure so far at
producing good crypto support in the lib.
phr> It's already the category king, because there are precisely zero
phr> other entrants in the category.

See my above references. Note, I don't use crypto at all, yet I was aware
of both of these (no Googling required).
The authors have not offered to contribute them, so they're not in the
category. The category consists of crypto modules that have actually
been offered. As I keep saying, I'd love it if someone else offered
one. I'm not eager for this headache. I just saw that somebody
really ought to do it, and nobody was doing it, so I decided I was
elected.
My guess would be they are substantially more mature than your
proposed module.
This may sound a little strange, since my module is not yet written,
but if we take maturity to mean lower future maintenance needs, then I
wouldn't say m2crypto is precisely more mature. m2crypto uses SWIG
and is intimately connected to an immensely complex package (OpenSSL)
anpd has to track new OpenSSL and SWIG releases. OpenSSL has all
kinds of OS dependencies, has high performance math subsystems written
in assembly code for various different cpu's, etc. So my guess is
that m2crypto has a longer development curve and will always need
significant amounts of maintenance. My module is more like a basic
portable math library that computes square roots, cosines, etc. and is
OS and CPU independent. Once written and tested, it's intended to be
pretty solid and not need much attention.

Andrew's package is along the same lines as mine, except Andrew's is
already done, so Andrew's is definitely more mature. I like to think
that my module's API incorporates some lessons from Andrew's and
improves on it, but that's not that important in practice.
phr> I read those as saying that no crypto module will be
phr> considered for inclusion whether or not it's the category
phr> king, because any such module might conflict with crypto
phr> regulations in some countries.

That may be a problem, sure. I'm not sure how the discussion here changes
that. That's just life as we know it.
It doesn't change it. However, it does change my level of interest in
writing a module, as is reasonable since the only important
characteristics of my module that are different from existing ones is
that mine is designed specifically to be suitable for the core and has
been offered for distribution in the core. Outside the core, the
existing modules work ok so I use them.
Why in the heck is inclusion in the standard library a requirement
for you to write this thing?
There's no other reason to write it. Outside the stdlib, there are
other modules that are already useable, like the ones you mentioned,
so I use those.
If it's useful to you, write it and get on with your life.
It's not worth doing unless it's intended for the stdlib. How many
times do I have to explain this?
phr> Tell me again how rotor got into the distribution.

Python's user community was probably a few hundred people. Guido
likely had no thoughts of world domination with the little language that
could. Times have changed.
If Python is aiming for world domination, it needs to be able to
compete with Java in security features. Java ships with the whole
JSSE system and a much fancier low level crypto architecture than the
one we've been discussing. So I think the Python folks should figure
out how to incorporate crypto without getting crushed under the iron
boot (or at least the wooden shoe) of the Dutch government or whatever
it is that they're worried about.
You seem think there was a PEP process and distutils and Wikis. I
suspect some of the algorithms one might include in a robust crypto
toolkit today weren't even invented in 1992.
AES wasn't invented then. DES became a US federal standard around 1977.
So throw away the rotor crutch and put your money where your mouth is.


If you mean write yet another crypto module knowing ahead of time that
it won't go into the stdlib, that would be more like throwing my money
down the drain.
Jul 18 '05 #62
phr
Skip Montanaro <sk**@pobox.com> writes:
>> As long as we are discussing cryptography, what's wrong with
>> m2crypto? Or, what about Andrew Kuchling's crypto toolkit?

Lucas> Umm, is it just me or did we just discuss the legal issues of
Lucas> that??

You may have. Whether or not there are legal issues with them is of little
relevance to the point I was making. Anything Paul writes would probably
have the same legal entanglements.


Andrew's toolkit does incorporate some patented algorithms, but those
aren't critical and could be removed from a stdlib version.
I was simply pointing out that maybe, just maybe, there are already
suitable candidates from a technical standpoint and that he doesn't
need to write anything.


There really don't appear to be any that are both technically
suitable, and that the authors are willing to contribute to PSF. I'd
be delighted to be wrong. Of course that still leaves the legal
issue.
Jul 18 '05 #63
ph*@localhost.localdomain wrote:
I don't see why you can't make up your mind enough to issue simple
statements like "the Python lib should have a module that does
so-and-so
I can say that assuming I know what so-and-so is. For the specific
case of AES, I would say "I don't think the Python lib necessarily
needs to have an AES module, but I would not object if it had one"
(the latter part in consideration of consequences that inclusion
of crypto code might have).
and it should meet such-and-such requirements
I can only say such things if I know such-and-such in detail
to specify requirements. For the specific case of AES, I don't
know enough about it to specify requirements. I will have to
trust others (and by that, I mean *multiple* others)
so if
someone submits one that meets the requirements and passes code review
and testing and doesn't have unexpected issues or otherwise fail to
meet reasonable expectations, we'll use it".
Because I cannot specify requirements, I cannot make such a promise.

In addition, for any new module, there is one primary requirement
for acceptance that cannot be fulfilled in code quality: the
contributor should promise to support the module in a foreseeable
future (i.e. a couple of years).
Again, we're talking about straightforward modules whose basic
interface needs are obvious.
You are talking about such a thing. I don't know enough about
the functionality to specify what an obvious interface is, or
to recognize one if I see it.
I don't know what OMG is, but there is no IETF requirement that any
implementations be available in any particular language.
See RFC 2026, section 4.1.2. Two independent implementations
are required for the document to advance to draft (!) standard.

However, the result of my not writing an AES module is that Python
doesn't have an AES module.
That's not true. PyCrypto does have AES support.
No, it's three steps
1. decide that you want to do it
2. do it
3. decide whether you are pleased with the result, and only
use it if you are

IOW, there should not be a blanket guarantee to use it after step 1.

But, it's completely normal to say before step 1 that "if the result
of step 2 does so-and-so, then I'll be pleased in step 3",


That's what I'm saying: If you distribute the module to users for
a year, and users express interest and support for your choice of
API, I'll support inclusion of the module. "do it" involves more than
just writing the code.
You and
Frederik seem to think there's something inappropriate or
self-inflated about wanting that expectation before committing to do a
pile of work that's primarily for other people's benefit.
It's very easy. If you are primarily do it for other people's
benefit, and if you don't find any satisfaction in the process
of doing it - THEN DON'T. I really mean that; this is how
free software works. People *volunteer* to do things. If they
don't volunteer - that's perfectly fine.
I think
your stated attitude is completely bizarre, that you can't really
believe anything so silly, so you're really just acting bureaucratic,
looking for excuses to say no instead of yes to worthwhile proposals.


As I said above - for the specific feature in question, I don't
care enough for the feature itself. Python will be just as useful
to me with the feature as it is without.

What I do care about is the effort that I will need to continue
maintaining Python. I don't want to have to maintain an ill-designed,
buggy module with no active maintainer, and I don't want to tell
people that I had to rip the module out just because it doesn't
work at all.

Regards,
Martin
Jul 18 '05 #64
"Martin v. Löwis" <ma****@v.loewis.de> writes:
I don't see why you can't make up your mind enough to issue simple
statements like "the Python lib should have a module that does
so-and-so
I can say that assuming I know what so-and-so is. For the specific
case of AES, I would say "I don't think the Python lib necessarily
needs to have an AES module, but I would not object if it had one"


Well, ok, you're changing your tune a little bit now, and getting more
reasonable. Before, you were making blanket statements of any modules
written for the Python stdlib. Now you're limiting it to AES and basing
it on some AES-specific things.
(the latter part in consideration of consequences that inclusion
of crypto code might have).
I would say the first thing I'd request would be your sense of how
plausible it is that the current no-crypto policy might be relaxed.
And I think this is about totally up to Guido, since (reading between
the lines of those python-dev messages) he's concerned about the Dutch
crypto restrictions, which makes some sense because he's Dutch and
still has to deal with the Dutch government from time to time, while
you or I might not care what the Dutch government thinks.

I was about to mention that Rop Gongrijp has been operating a crypto
company in Holland (www.nah6.com) but I now see that it's moved to
Germany, and I wonder if those Dutch restrictions might be the reason.
I don't know any details though.
and it should meet such-and-such requirements


I can only say such things if I know such-and-such in detail
to specify requirements. For the specific case of AES, I don't
know enough about it to specify requirements. I will have to
trust others (and by that, I mean *multiple* others)


OK. If it helps, I can tell you that the Python crypto crowd hangs
out on a mailing list called python-crypto and has discussed this
module at some length.
so if someone submits one that meets the requirements and passes
code review and testing and doesn't have unexpected issues or
otherwise fail to meet reasonable expectations, we'll use it".


Because I cannot specify requirements, I cannot make such a promise.


Again OK. I had thought from earlier discussions that you were a
crypto developer and familiar with this stuff; no problem if you're
not. However, in that case, you probably wouldn't be using the module
if it got included. I think the best policy if you don't feel
comfortable supporting including some module (whether it's crypto or
something else) that you're not personally going to use, is don't
support inclusion, but also don't obstruct it. If there's some other
stdlib maintainers who are knowledgeable about that kind of module and
have an opinion about it, then go along with them unless there's a
clear reason not to. Like, if I were a stdlib maintainer and somebody
submitted a fancy multigrid PDE solver, that's outside my expertise
and I'd go along with whatever Tim recommended.
In addition, for any new module, there is one primary requirement
for acceptance that cannot be fulfilled in code quality: the
contributor should promise to support the module in a foreseeable
future (i.e. a couple of years).
That's not too unreasonable, though I don't think I've heard it
mentioned as a requirement before.
You are talking about such a thing. I don't know enough about the
functionality to specify what an obvious interface is, or to
recognize one if I see it.
The thing to do then is just defer the whole issue to someone on the
core team who uses that type of function. Only if no one on the team
has an opinion do you start having to look for stuff like wide use of
the module in the outside community.
I don't know what OMG is, but there is no IETF requirement that any
implementations be available in any particular language.


See RFC 2026, section 4.1.2. Two independent implementations
are required for the document to advance to draft (!) standard.


That says nothing about the implementation languages. Both might be
in Python, or one might be in VHDL and the other in Ada, or whatever.

This particular module we're discussing is a straightforward
implementation of AES, DES, and FIPS 81, for which there are thousands
of implementations in both hardware and software. For the specific
Python API, there's a Python implementation and the proposed new
module is a C implementation that does the exact same thing only
faster. So that's two implementations, though not independent in the
IETF sense, because they'd both be written by the same person.

I've never heard of any requirement before that there be two separate
implementations of every Python stdlib module, by the way. That would
be silly.
However, the result of my not writing an AES module is that Python
doesn't have an AES module.


That's not true. PyCrypto does have AES support.


That's not in any Python distro that I know of. I'm talking about
modules in the stdlib, not external modules. "Batteries included",
you know.
That's what I'm saying: If you distribute the module to users for
a year, and users express interest and support for your choice of
API, I'll support inclusion of the module. "do it" involves more than
just writing the code.
Well, that's nice of you to offer to support inclusion of a module
that you're not likely to use yourself and whose functionality you
don't really understand (I'm not being sarcastic). If you're going to
offer such support it makes sense to impose an unusually high standard
for offering it. I myself would probably never support including any
such module no matter how long it was distributed, but rather would
just defer the whole question to people experienced with such modules
and trust their sense of what the acceptance criteria should be for a
specific module. That is, I'd abstain from any decision.
It's very easy. If you are primarily do it for other people's
benefit, and if you don't find any satisfaction in the process
of doing it - THEN DON'T. I really mean that; this is how
free software works. People *volunteer* to do things. If they
don't volunteer - that's perfectly fine.
That doesn't map well onto the real world. Say I'm a professional
cook and I have a good recipe for spam, eggs, sausage, and spam. I
write the PyCon organizers and ask if they'd like me to cook up a big
batch to serve for breakfast for PyCon attendees (this is as a
volunteer, all ingredients provided at my own expense, etc).
Reasonable answers might be:

1) No, we don't want to serve breakfast at PyCon, we thought about
it but decided against it. => OK, fine, I don't cook it.

2) Hey, that sounds good--cook enough for 500 people and bring it to
the site before the conference starts, and we'll put it on the
menu unless it's below expectations. => OK, fine, I start
cooking. Nobody has made an ironclad promise, but the organizers
have told me their basic intentions, which is good enough for me.

3) Hmm, that recipe sounds interesting, why don't you send one or
two portions to the organizers, we'll try them and let you know.
=> OK, this is basically a prototype implementation, like the
pure-Python version of the AES module, that's fine for testing
but not fast enough to serve (e.g.) 500 web connections. I
send the small batch and wait for an answer before deciding to
start the much bigger job of making a 500-person batch.

Your answer is more like "make enough for 500 people and bring it to
the conference site and only then will we even THINK about whether to
serve it for breakfast. Serving it at PyCon shouldn't matter to you,
what are you, some kind of egotist? You like cooking, so if you make
the 500 portions and then the organizers decline them, you can always
hand them out in the street. If you don't find any satisfaction in
that, THEN DON'T DO IT."

If the cook's goal is not just to feed Python users in the street, but
to actually improve PyCon itself by providing for PyCon to serve
breakfast to attendees, I hope you can see why he might not think much
of your answer.
As I said above - for the specific feature in question, I don't
care enough for the feature itself. Python will be just as useful
to me with the feature as it is without.


I'm sure that's true of lots of modules. If you're the sole maintainer
of the Python distro, then you have to evaluate every module that anyone
submits. But since you're part of a reasonable sized group, I think
it's best to examine the modules you care about, but don't feel that
you have to have your hands in everything.
Jul 18 '05 #65
Paul Rubin wrote:
I can say that assuming I know what so-and-so is. For the specific
case of AES, I would say "I don't think the Python lib necessarily
needs to have an AES module, but I would not object if it had one"

Well, ok, you're changing your tune a little bit now, and getting more
reasonable. Before, you were making blanket statements of any modules
written for the Python stdlib. Now you're limiting it to AES and basing
it on some AES-specific things.


And I still stand by those blanket statements. Any new module (i.e.
specific code) should see real users for some time before it can
be incorporated into Python.

The question whether I would like to see a certain *feature*
in Python (whether as a separate module, a language feature, or
otherwise) is an entirely different question. That I would not
object to an AES code in general doesn't imply that I will
agree to any AES module that somebody contributes.
I would say the first thing I'd request would be your sense of how
plausible it is that the current no-crypto policy might be relaxed.
There is no such policy in force. Different people have different
opinions, and those opinions change over time. I cannot speak for
others.
Again OK. I had thought from earlier discussions that you were a
crypto developer and familiar with this stuff; no problem if you're
not. However, in that case, you probably wouldn't be using the module
if it got included. I think the best policy if you don't feel
comfortable supporting including some module (whether it's crypto or
something else) that you're not personally going to use, is don't
support inclusion, but also don't obstruct it.
Thanks for the advise, but I'll have my own policies. I have included
several modules in the past which I'm not personally using.
In addition, for any new module, there is one primary requirement
for acceptance that cannot be fulfilled in code quality: the
contributor should promise to support the module in a foreseeable
future (i.e. a couple of years).

That's not too unreasonable, though I don't think I've heard it
mentioned as a requirement before.


See PEP 2.
I've never heard of any requirement before that there be two separate
implementations of every Python stdlib module, by the way. That would
be silly.
And I did not suggest such a thing. You said you never heard about
a process which requires an implementation before deciding that the
proposed feature is good, and I said that, for an example, IETF even
requires *two* implementations before deciding that the feature is
good.
However, the result of my not writing an AES module is that Python
doesn't have an AES module.


That's not true. PyCrypto does have AES support.


That's not in any Python distro that I know of.


Sure, but "Python" is more than the Python core.
I myself would probably never support including any
such module no matter how long it was distributed, but rather would
just defer the whole question to people experienced with such modules
and trust their sense of what the acceptance criteria should be for a
specific module. That is, I'd abstain from any decision.
With that attitude, the patches tracker on SF would grow unbounded,
because we *continuously* get patches that no core developer has
any personal interest in. I'll try to look at *all* patches, whenever
I can, and in the process, I have to learn about technologies which
I've never used before (primarily operating systems, but also
APIs and protocols).

Rejecting all these patches would be unfair to the contributors.
Your answer is more like "make enough for 500 people and bring it to
the conference site and only then will we even THINK about whether to
serve it for breakfast.
I cannot see these as the same thing.
I'm sure that's true of lots of modules. If you're the sole maintainer
of the Python distro, then you have to evaluate every module that anyone
submits. But since you're part of a reasonable sized group, I think
it's best to examine the modules you care about, but don't feel that
you have to have your hands in everything.


Thanks again for the advise, but this is not how Python is being
maintained.

Regards,
Martin
Jul 18 '05 #66
"Martin v. Löwis" <ma****@v.loewis.de> writes:
And I still stand by those blanket statements. Any new module (i.e.
specific code) should see real users for some time before it can
be incorporated into Python.
Let's see, the urandom module was recently released in 2.4, I think
initially at my urging. I believe it was written just for 2.4,
combining a Windows function used only (as far as I know) by its
author, and a wrapper to read /dev/urandom on Posix systems. It
didn't have any wide distribution or use before incorporation, and
didn't need it. It was simple enough that a reasonable code review
and some testing on the relevant platforms was enough to make make
sure it worked properly.
I would say the first thing I'd request would be your sense of how
plausible it is that the current no-crypto policy might be relaxed.


There is no such policy in force. Different people have different
opinions, and those opinions change over time. I cannot speak for others.


If you can't speak for others, how can you say there's no policy in
force? If Guido says "no crypto", is that something other than a
policy? And are you speaking for him when you say that? <wink>
proposed feature is good, and I said that, for an example, IETF even
requires *two* implementations before deciding that the feature is good.
No, that's completely wrong. IETF requires two implementations before
publishing a specification--a precise description--as a draft
standard, which means agreeing on how the feature should work at the
finest level of detail. That's much different than merely deciding
that a feature is good and inviting people to work on it.

IETF often decides and announces that a feature is good long before
any such details are decided or agreed on. For example, they decided
that IP connection encryption was a good feature, and appointed a
working group a general mandate to go figure out the details (IPSEC)
and write a spec. See? IETF announced in advance that IPSEC was a
good feature, and the working group members knew of that decision
before they committed the resources needed to write the draft. And
when they started, there were zero actual IPSEC implementations,
because nobody knew what precise characteristics IPSEC would
eventually have. The implementations got written in the process of
developing the spec and presenting it as a standard.

The two-implementation policy means there has to be two
implementations before the draft is finally declared complete, not
before the IETF can announce that it wants a standard to be written
and intends to use whatever the working group comes up with once the
draft and two implementations are done. That is what happens in any
normal professional process. Nobody commits resources to develop
something (other than for experimentation) unless they think the end
result will actually be used. Are you really having trouble
understanding this?
That's not true. PyCrypto does have AES support.

That's not in any Python distro that I know of.


Sure, but "Python" is more than the Python core.


Tell me again what "batteries included" means. I personally interpret
"included" as meaning "included". That means when I install Python,
the module is right there in the distro. So if it's not in the
distro, Python itself doesn't have it. "Batteries included" is an
expression of Pythonic philosophy that says: putting components that
end-users need in the distro is superior to making the users get them
from somewhere else separately. I realize that the difference might
not matter to you, but it does matter to a heck of a lot of other
users.
With that attitude, the patches tracker on SF would grow unbounded,
because we *continuously* get patches that no core developer has
any personal interest in.


Obviously those require a different type of consideration. I'm
talking about patches where there's a core developer with an interest.
Let's say the cmath module doesn't exist and someone submits it. It
computes functions like cos(z) for complex arguments. If Tim, the
numerics expert, then says "this is worth having and I've examined the
code and it looks good, let's put it in", I'd say that takes care of
it, and I think that's what typically happens with real modules. If
nobody on the core team has an informed opinion, then it has to be
decided some slower and more formal way.
Jul 18 '05 #67
On Tuesday 25 January 2005 01:51 am, Fredrik Lundh wrote:
ah, you're the guy who flamed me and called me names because PIL didn't
fit some hypothetical GNU-inspired definition of "open source software." it's
always sad when people have to attack those who don't share their religion,
but it's not that much I can do about that.


That is a complete misrepresentation of the post in question. It was in
no manner of interpretation a "flame". You simply took offense out of thin
air. And there's not much *I* can do about *that*.

I figured that my intent to contribute to your project was sufficient to
indicate my good intentions.
So, naturally, I just dropped it.


which means that you have no idea how easy or hard it is to install PIL today
(hint: things have changed)


Oh? In which version?

Are you saying that your installation procedure now honors the "--with-jpeg=" and
"--with-png=" flags to configure? Because although you had these flags, in the last
version I installed from source, they did not work -- it was still necessary to edit
the Makefile.

That was either 1.1.3 or 1.1.4, and I'm pretty sure it was the latter.

Now if you've actually fixed this, I look forward to trying it out the next
time I have to do a complete from-source installation. I'll say "thank you",
just in case.

Cheers,
Terry

--
--
Terry Hancock ( hancock at anansispaceworks.com )
Anansi Spaceworks http://www.anansispaceworks.com

Jul 18 '05 #68
Paul Rubin wrote:
Let's see, the urandom module was recently released in 2.4, I think
initially at my urging.
There is no urandom module in Python 2.4.
If you can't speak for others, how can you say there's no policy in
force?
I should say I'm not aware of a policy.
If Guido says "no crypto", is that something other than a
policy? And are you speaking for him when you say that? <wink>
If he had said such a thing in general, it would be a policy
(although it best would be documented somewhere). I don't think
he said that, in general, and with the intent of stating a policy.
That's much different than merely deciding
that a feature is good and inviting people to work on it.
We don't normally invite people to work on anything. People
pick the things they work on themselves.
IETF often decides and announces that a feature is good long before
any such details are decided or agreed on. For example, they decided
that IP connection encryption was a good feature, and appointed a
working group a general mandate to go figure out the details (IPSEC)
and write a spec. See?
No. They decided that spam machinery in DNS would be a good thing,
people started to work on it, and then they decided that it is not
such a good thing, after all, because it causes too many problems.

So the decision "this is a good thing" is no guarantee for
"if it is done, we will publish it as a standard". They might
start a process, people might get nearly through it, and then
the process stucks or is given up entirely. Happens all the time.
I realize that the difference might
not matter to you, but it does matter to a heck of a lot of other
users.
It does matter for me, yet Python is still more than the core.

You might be ignoring that, but it surely is more to the many
developers which create Python libraries and distribute them
themselves, see

http://www.python.org/pypi

THIS IS ALL PYTHON.
Obviously those require a different type of consideration. I'm
talking about patches where there's a core developer with an interest.


I though you were talking about the AES module...

Regards,
Martin
Jul 18 '05 #69
"Martin v. Löwis" <ma****@v.loewis.de> writes:
Let's see, the urandom module was recently released in 2.4, I think
initially at my urging.
There is no urandom module in Python 2.4.


Oops, sorry, it's in the os module:

http://docs.python.org/lib/os-miscfunc.html

The difference is simply a matter of the packaging. It's important
functionality and it was added after an amount of review and testing
that was deemed to be reasonable. Unless you're saying that if I
wanted to add AES to the string module (so you could say
'spam and sausage'.aes_encrypt('swordfish banana')) instead of writing a
separate module, then we wouldn't need this discussion. (Putting it
in the string module is not a serious proposal but it wouldn't be
completely insane.) What matters is the code complexity, not whether
something is in a separate module or not.
If Guido says "no crypto", is that something other than a
policy? And are you speaking for him when you say that? <wink>


If he had said such a thing in general, it would be a policy
(although it best would be documented somewhere). I don't think
he said that, in general, and with the intent of stating a policy.


Well, if he indicates that it's not a policy and that the question is
still open, then I could see getting interested again in writing an
AES module. At the moment I continue to see his python-dev post as
quite discouraging.
We don't normally invite people to work on anything. People pick the
things they work on themselves.
Not true. For example, you once invited me to work on an ancillary
message feature for the socket module (SF bug 814689), and so it's
been on my want-to-do-one-of-these-days list since then. I think it's
reasonable for me to have taken your message there as an expression of
interest, sufficient to get me to want to work on it. So it's bogus
to say the Python developers should avoid expressing interest in
something that hasn't already been written.
IETF often decides and announces that a feature is good long before
any such details are decided or agreed on.


No. They decided that spam machinery in DNS would be a good thing,
people started to work on it, and then they decided that it is not
such a good thing, after all, because it causes too many problems.


There's no contradiction. IETF decided something based on some
reasonable beliefs that later turned out to be wrong, so they had to
undo the original decision because the original expectations couldn't
be met. That happens sometimes. Everyone has worked on projects that
ended up failing and getting cancelled for reasons like that. There
are few ironclad guarantees of anything, only reasonable beliefs. But
real-world organizations are not afraid to start projects based on
reasonable beliefs regardless. Otherwise, few things would ever get
done.
It does matter for me, yet Python is still more than the core.

You might be ignoring that, but it surely is more to the many
developers which create Python libraries and distribute them
themselves, see

http://www.python.org/pypi
THIS IS ALL PYTHON.


No. Those are programs people have written in Python or as Python
extensions. I've written programs like that myself. I don't consider
them to be part of Python. They are separate programs. If one of
them gets accepted into the distro (as this thread is about), then it
becomes part of Python. See <http://www.python.org/doc/faq/general.html>:

1.1.6 How do I obtain a copy of the Python source?

The latest Python source distribution is always available from
python.org, at http://www.python.org/download/. The latest development
sources can be obtained via anonymous CVS from SourceForge, at
http://www.sourceforge.net/projects/python.

Python is what you get from following the download instructions and
installing the result. In <http://www.python.org/doc/faq/installed.html>,
where it asks "why is Python installed on my machine?", that is what
they are referring to by "Python".
Obviously those require a different type of consideration. I'm
talking about patches where there's a core developer with an interest.


I though you were talking about the AES module...


No, when this started you were talking about modules in general.
Jul 18 '05 #70
Paul Rubin wrote:
Oops, sorry, it's in the os module:

http://docs.python.org/lib/os-miscfunc.html

The difference is simply a matter of the packaging.
No, it's not. It also is a matter of code size, and impact. Small
additions can be reviewed and studied more easily, and need to be
tested on less users. A new module is on a larger scale than
a mere new function.
Unless you're saying that if I
wanted to add AES to the string module (so you could say
'spam and sausage'.aes_encrypt('swordfish banana')) instead of writing a
separate module, then we wouldn't need this discussion.
Indeed, if it was a single new function to an existing module, I would
not require that this be delivered to users first. It is entire new
libraries that I worry about.

If you would propose a change to the string module to add an aes_encrypt
function, I would immediately reject that patch, of course, because that
function does not belong to the string module.
What matters is the code complexity, not whether
something is in a separate module or not.
A module *is* typically more complex than a single function. If
your new module has only a single new function, we should discuss
whether it really needs to be a separate module.
Well, if he indicates that it's not a policy and that the question is
still open, then I could see getting interested again in writing an
AES module. At the moment I continue to see his python-dev post as
quite discouraging.
And, again, I consider this perfectly fine. This would be a volunteer
effort, and volunteers are free to work on whatever they please.

Furthermore, people who want an AES module for Python could get one
from

http://sourceforge.net/projects/cryptkit/

Maybe Bryan Mongeau will contribute this code to Python some day.
Not true. For example, you once invited me to work on an ancillary
message feature for the socket module (SF bug 814689), and so it's
been on my want-to-do-one-of-these-days list since then. I think it's
reasonable for me to have taken your message there as an expression of
interest, sufficient to get me to want to work on it. So it's bogus
to say the Python developers should avoid expressing interest in
something that hasn't already been written.


I did not say that. I said we don't normally invite people to work
on anything - I said not that we *should* not invite them. Now that
you mention it, I find that there is an important exception from my
factual statement: I do regularly ask people reporting bugs
or requesting features to work fixing the bugs or implementing the
features. It is perfectly fine if they say "no" then. If they say
yes, there is an implied promise that I'll review their code when
they are done.

As it appears to be clear that you are not going to implement
an AES module in the foreseeable future, and as it also seems
to be clear that you cannot talk me into changing my views on
how Python should be developed, I think further discussing this
entire thing is pointless.

Regards,
Martin
Jul 18 '05 #71
"Martin v. Löwis" <ma****@v.loewis.de> writes:
Indeed, if it was a single new function to an existing module, I would
not require that this be delivered to users first. It is entire new
libraries that I worry about.
Why is it different if a single new function is added to an existing
module, or if the single new function has the boilerplate of a new
module wrapped around it?

Look at the sha and md5 modules. They are very similar in both
interface and implementation. The only internal function that's
really different is the update operation; they actually might have
been combined into one module that did the other operations with the
same code. But, it's also reasonable to have them as separate
modules. If users start needing sha256, it could be done the same
way, one new update operation and the rest boilerplate, but in
practice it would probably be a separate module.

Are you saying if there was user demand for an sha256 module and
someone wrote one, you'd still require a year of separate distribution?
A module *is* typically more complex than a single function. If
your new module has only a single new function, we should discuss
whether it really needs to be a separate module.
I previously had the mistaken belief that urandom was a new module
rather than a function inserted into an existing module. Note that
the urandom's implementation is not ultra-trivial.

An AES or DES addition to an existing module that implements just one
call:
ECB(key, data, direction)
would be a huge improvement over what we have now. A more complete
crypto module would have some additional operations, but ECB is the
only one that's really essential. I already have a pure-Python module
that does all the other operations using ECB as a subroutine.
It's speed isn't great but it's useable in some real applications.
It's only the ECB operation that's intolerably slow in Python.

If you think a function like that could be added to some existing
module with less hassle than adding a new module, then I can write one
and submit it.
Well, if he indicates that it's not a policy and that the question is
still open, then I could see getting interested again in writing an
AES module. At the moment I continue to see his python-dev post as
quite discouraging.


And, again, I consider this perfectly fine. This would be a volunteer
effort, and volunteers are free to work on whatever they please.


Well, volunteers are more likely to work on modules that are mentioned
as being welcome by the developers, than modules affected by explicit
prior developers' public decisions that cast a chill over the hope of
ever getting such a module accepted.
Furthermore, people who want an AES module for Python could get one from
Come on, you're being deliberately obtuse, we've discussed this over
and over. There are plenty of AES modules that people can get from
somewhere. The topic is what it takes to have an AES module that
people don't NEED to get from anywhere, because they already have it
from having Python installed. Do I have to keep repeating "batteries
included" until you understand what it means?
http://sourceforge.net/projects/cryptkit/

Maybe Bryan Mongeau will contribute this code to Python some day.
Well, that code has been around for over a year, people are using it,
etc. Are you saying you'll support its inclusion if Bryan offers to
contribute it? I've examined that module, I wouldn't consider it
ideal for the core (besides AES, it has some complicated additional
functions that aren't useful to most people), but it would certainly
take care of my AES needs (it's apparently missing DES though).
I did not say that. I said we don't normally invite people to work
on anything - I said not that we *should* not invite them.
I would say that inviting people to work on a module for the stdlib
means the developers have thought about whether such a module would be
useful and worth including, and are indicating that they're favorable
to the idea. However, you wrote:

In Message-ID: <41**********************@news.freenet.de>
So if the module was primarily written to be included in the core, I
would initially reject it for that very reason. After one year or so
in its life, and a recognizable user base, inclusion can be considered.

The context was new modules in general, not specifically an AES
module. Since "considered" means "thought about", so you said
inclusion shouldn't even be thought about until the module is already
done. That's completely in conflict with the idea of inviting anyone
to work on a new module, since inviting means that there's been some
thought.
Now that you mention it, I find that there is an important exception
from my factual statement: I do regularly ask people reporting bugs
or requesting features to work fixing the bugs or implementing the
features. It is perfectly fine if they say "no" then. If they say
yes, there is an implied promise that I'll review their code when
they are done.
I would say there's an implied promise of something more than a code
review. There's an implied statement that you agree that the proposed
new functionality is useful, which means the patch has a good chance
of being accepted to the stdlib if it's not too messy or cumbersome.
That's a heck of a lot different from saying "why don't you write that
patch and distribute it independently for a year on a purely
speculative basis, and then I'll think about whether it's worthwhile
to include it or not".
As it appears to be clear that you are not going to implement an AES
module in the foreseeable future,
The reason for that is as far as I can tell, even if I follow 100% of
your prescription of writing the module, releasing it independently
and supporting it for a year, and then offering to contribute it
complete with favorable user reviews and a promise of two years of
further support, the probability of it being accepted and included is
still close to zero. In more recent messages you've suggested that my
reading of that probability is wrong and that it's actually higher
than zero.

So let me just ask you one final question: suppose I do all that
stuff. The question: in your personal opinion, based on the best
information you have, what is the your own subjective estimate of the
probability?

I won't say I'd immediately replace my estimate with yours, but if you
name a reasonably high number and tell me that you really believe that
number, then I could get interested again.
and as it also seems to be clear that you cannot talk me into
changing my views on how Python should be developed, I think further
discussing this entire thing is pointless.


Well, you can have whatever views you want, but one I've thing I've
realized from this thread is that many of the frustrations I often
encounter with Python are a direct result of a development process
that's successful in some ways but dysfunctional in others.
Jul 18 '05 #72
http://www.python.org/pypi
THIS IS ALL PYTHON.


Paul> No. Those are programs people have written in Python or as Python
Paul> extensions.

What's your point? That I have to download and perhaps install them to use
them? In that case, how are these two scenarios different:

* I have to download and build the MySQLdb package to talk to MySQL
servers from Python code

* I have to ensure that the readline library and include files are
installed on my system before the readline module (which is included
in the core distribution) can be built

I and many other people happily use external packages other people have
written as well as make stuff available. My guess is that you do as well.
If everyone adopted your position that it wasn't Python unless it had been
added to the core, we'd all be reinventing lots of wheels or tackling much
less challenging tasks, if we programmed in Python at all. Here's an
incomplete list of stuff not in the core I have used happily over the past
several years to do my jobs using Python:

* MySQLdb, Sqlite, pycopg, sybase-python - all database modules
* CSV, Object Craft's csv, DSV - csv modules predating csv in the core
* SpamBayes
* Quixote
* Docutils
* MoinMoin
* Pyrex
* Psyco
* PyInline
* PyGTK
* xmlrpclib before it was in the core
* MAL's mx.DateTime before the core datetime module was available
* timeout_socket before sockets supported timeouts

Many of those things I could never have written myself, either for lack of
time, lack of skill or both. I'm grateful they were available when I needed
them and feel no qualms about using them even though they are not
distributed with Python proper.

Notice another interesting feature of several of those items: csv,
xmlrpclib, mx.DateTime, timeout_socket. They were all modules I used that
eventually wound up in the core in some fashion. They didn't go in the core
first, then demonstrate their usefulness. It was the other way around.

Not everything that is useful belongs in the core distribution. I think you
are confusing "batteries included" with "everything, including the kitchen
sink".

Skip
Jul 18 '05 #73
What matters is the code complexity, not whether something is in a
separate module or not.


Martin> A module *is* typically more complex than a single function.

And one that deals with cryptography is likely to be even more complex.

Skip
Jul 18 '05 #74
Skip Montanaro <sk**@pobox.com> writes:
What's your point? That I have to download and perhaps install them to use
them? In that case, how are these two scenarios different:

* I have to download and build the MySQLdb package to talk to MySQL
servers from Python code

* I have to ensure that the readline library and include files are
installed on my system before the readline module (which is included
in the core distribution) can be built
The difference is that once Python is installed on your machine and
you can get a ">>>" prompt, you have readline available right away but
you have to download something to use MySQLdb. Whoever took care of
your Python installation, and it may not have been you, also took care
of readline. The past several OS distributions I've installed have
included Python and readline out of the box, so I never had to think
about readline. The last time I used a Python instance that didn't
come with the OS (on Windows XP at work), the IT department had
installed Python on my desktop before I started using it, so I still
didn't have to worry about readline. But any module that doesn't
come in the distro, I have to download myself.
I and many other people happily use external packages other people have
written as well as make stuff available. My guess is that you do as well.
No, I don't. I do use them sometimes but I'm unhappy about them. If
I can write something using a core module instead of an external
module, I prefer to use the core module. So I'll generally use dbm
instead of MySQL unless I really need MySQL, which I haven't yet in
Python (I've used MySQL with Perl dbi, but Perl, you know, shudder).

Also, external module installation scripts often don't work properly,
so I end up having to wrestle the code to get it installed. And if a
geek like me has such trouble installing external modules, what hope
does a normal end-user have? Maybe if you're using Windows, that
stuff has been debugged better, but I've had poor results under
GNU/Linux.

I've had this discussion here before, maybe not with you. What I
really want is zero installations of anything. I just want to go to
the store and buy a new computer and have a complete OS install with
full sources and a full set of applications including Python already
installed when I first power it up. My practical approximation is to
buy a new computer, immediately reformat the HD to remove the icky
Redmond virus, and then install a GNU/Linux distro that includes
Python (and readline). If Python really aims for world domination,
that means it has to shoot for being preinstalled on almost every new
computer the way Windows is now. And all the interesting modules
should be there, maybe in a "contrib" directory that gets little or no
maintenance priority from the core team.
If everyone adopted your position that it wasn't Python unless it
had been added to the core, we'd all be reinventing lots of wheels
or tackling much less challenging tasks, if we programmed in Python
at all. Here's an incomplete list of stuff not in the core I have
used happily over the past several years to do my jobs using Python:
That makes no sense at all. That list is a list of programs written
in the Python language. They are Python programs, where Python is an
adjective. Python, the noun referring to a piece of software,
generally means the stuff in the Python distro. That doesn't stop
programs outside the distro from being useful. Mediawiki is a PHP
program. That doesn't mean Mediawiki is part of PHP.
* MySQLdb, Sqlite, pycopg, sybase-python - all database modules
These should all be in the core if Python wants to be a serious
competitor to PHP, which comes with interfaces for those db's and
several additional ones besides. That these modules are missing are a
significant library deficiency.
* CSV, Object Craft's csv, DSV - csv modules predating csv in the core
That's fixed now, csv is in the core.
* SpamBayes
I have the impression this is an application and not a module, or
anyway is written mainly to support one application. Should be
separate. Also, it's written in Python(?) rather than C, which means
the installation headaches from not being in the core aren't so bad.
* Quixote
Don't know what this is.
* Docutils
Should be in the core if it's what I think it is.
* MoinMoin
Application, should be separate. Also, GPL'd, I think. Can't be
distributed under PSF license.
* Pyrex
Sort of problematic, would be interesting to have something like this
in the core but maybe Pyrex as it currently stands isn't the answer.

I have the impression that PyPy is going to depend on Pyrex in a
fundamental way, so it will have to be in the core when we dump CPython.
* Psyco
I think this isn't ready for prime time yet. Should go into the core
once it is.
* PyInline
Not sure what this is.
* PyGTK
wxPython might be a better choice. wxpython.org quotes Guido as
saying "wxPython is the best and most mature cross-platform GUI
toolkit, given a number of constraints. The only reason wxPython isn't
the standard Python GUI toolkit is that Tkinter was there first". I
don't have direct experience with either wxPython or PyGTK, though.
If I can get by with Tkinter, I'd rather do that, since Tkinter is in
the core.
* xmlrpclib before it was in the core
1. Did you really need this, instead of some more reasonable rpc format?
xdrlib has been in the core forever.
2. Isn't xmlrpclib written in Python?
* MAL's mx.DateTime before the core datetime module was available
See, as Python improved, those things went into the core.
* timeout_socket before sockets supported timeouts
Could you use sigalarm instead?
Many of those things I could never have written myself, either for
lack of time, lack of skill or both. I'm grateful they were
available when I needed them and feel no qualms about using them
even though they are not distributed with Python proper.
Sure, it's fine if you have all those modules and you write a Python
program that uses, say, five of them. External modules aren't so bad
when the developer and the end user are the same person. What happens
if you send your Python program to a nonprogrammer friend who has just
a vanilla Python installation? Now he has to download and install
those five modules too. You send him the url's where you got the
modules a year ago. What are the chances that the 5 url's even all
still work, much less the chance of him being able to install and run
all 5 of the modules without needing help? What if the versions he
downloads (from separate developers) have gotten out of sync with each
other and can't interoperate any more?

If Python's maintainers are going to recommend those modules to people
as a substitute for providing those functions in the core, it would be
a big help if the modules were mirrored on python.org instead of
merely linked, since a lot of links turn into 404's over time.
Notice another interesting feature of several of those items: csv,
xmlrpclib, mx.DateTime, timeout_socket. They were all modules I used that
eventually wound up in the core in some fashion. They didn't go in the core
first, then demonstrate their usefulness. It was the other way around.
I'm not sure about timeout_socket and it sounds like it should have
just been a patch to the socket module, not a new module. csv is
quite a complex module and took a lot of tweaking and PEP editing
before standardization. But the need for it was obvious; the only
good reason it wasn't in the core ages ago was that no one had done
the work of writing it and shaking it out. xmlrpclib, not sure. How
long was it in separate distribution? Also, xmlrpc is a pretty new
protocol so it took a while before people wanted it. DES has been
around since the 1970's and AES has about the same users as DES, so
there's a known level of demand. That should be enough to say yes or
no to whether there should be a core module or not.

Also, your notion of trying to create a "category king" of AES modules
doesn't reflect how these things work. There are at least 10
different AES modules that provide the same AES function. If somebody
is using one, they have no reason to switch to another. If it takes
20 visible users for including a module to be worthwhile, and those 10
modules have 5 users each, those populations are going to stay stable
until one of them goes in the core and becomes the standard. (And
actually: mxCrypto is the most capable of these packages and might be
the one with the most users, but it's completely unsuitable for the
core because of its size).
Not everything that is useful belongs in the core distribution. I
think you are confusing "batteries included" with "everything,
including the kitchen sink".


Well, if you compare the Python stdlib with the toolkits that come
with competing languages, say PHP or Java, you can see that Python's
lib could be enhanced with considerably more stuff without being
excessive.
Jul 18 '05 #75
Skip Montanaro <sk**@pobox.com> writes:
And one that deals with cryptography is likely to be even more complex.


No. The AES module would have about the same complexity as the SHA module.
Jul 18 '05 #76
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
actually: mxCrypto is the most capable of these packages and might be
the one with the most users, but it's completely unsuitable for the
core because of its size).


Oops, I should say, mxCrypto itself isn't that large; the issue is
that it needs OpenSSL which is a big unwieldy program. Having
mxCrypto in the core as an OpenSSL interface is a legitimate notion.
But there should be something that doesn't depend on OpenSSL.
Jul 18 '05 #77
Paul Rubin wrote:
An AES or DES addition to an existing module that implements just one
call:
ECB(key, data, direction)
would be a huge improvement over what we have now.
Apparently, people disagree on what precisely the API should be. E.g.
cryptkit has

obj = aes(key)
obj.encrypt(data)

I think I would prefer explicit encrypt/decrypt methods over a
direction parameter. Whether or not selection of mode is a separate
parameter, or a separate method, might be debatable - I'ld personally
prefer a separate method. However, we would have to ask users.
If you think a function like that could be added to some existing
module with less hassle than adding a new module, then I can write one
and submit it.
I would trust my intuition more for a single function than for an
entire API. In this specific proposal, I think I would trust my
intuition and reject the ECB function because of the direction
argument.
Come on, you're being deliberately obtuse, we've discussed this over
and over. There are plenty of AES modules that people can get from
somewhere. The topic is what it takes to have an AES module that
people don't NEED to get from anywhere, because they already have it
from having Python installed. Do I have to keep repeating "batteries
included" until you understand what it means?
I fully understand what you desire - to include the module "as a
battery". What makes this decision difficult is that you fail to
understand that I don't want included batteries so much that I
would accept empty or leaking batteries.
http://sourceforge.net/projects/cryptkit/

Well, that code has been around for over a year, people are using it,
etc. Are you saying you'll support its inclusion if Bryan offers to
contribute it?


*Now* you get it. Precisely that. I would ask the users what they
think about the API (shouldn't be too difficult because the module
does have users) and what they think about other aspects (performance,
stability, and so on).
I've examined that module, I wouldn't consider it
ideal for the core (besides AES, it has some complicated additional
functions that aren't useful to most people)
Ok, that would be a problem. If this is a simple removal of functions
that you'ld request (which functions?), I'ld try to collect opinions
on that specific issue, and ask Bryan whether he could accept
removal of these functions.
So if the module was primarily written to be included in the core, I
would initially reject it for that very reason. After one year or so
in its life, and a recognizable user base, inclusion can be considered.

The context was new modules in general, not specifically an AES
module. Since "considered" means "thought about", so you said
inclusion shouldn't even be thought about until the module is already
done. That's completely in conflict with the idea of inviting anyone
to work on a new module, since inviting means that there's been some
thought.
I rarely invite people to work on new modules. For new modules, I
normally propose that they develop the module, and ship it to users
for some time.

I may have made exceptions to this rule in the past, e.g. when the
proposal is to simply wrap an existing C API in a Python module
(like shadow passwords). In this case, both the interface and
the implementation are straight-forward, and I expect no surprises.
For an AES module (or most other modules), I do expect surprises.
I would say there's an implied promise of something more than a code
review. There's an implied statement that you agree that the proposed
new functionality is useful, which means the patch has a good chance
of being accepted to the stdlib if it's not too messy or cumbersome.
I have said many times that I am in favour of including an AES
implementation in the Python distribution, e.g. in

http://mail.python.org/pipermail/pyt...il/034963.html

What I cannot promise is to include *your* AES implementation,
not without getting user feedback first. The whole notion of
creating the module from scratch just to include it in the core
strikes me as odd - when there are so many AES implementations
out there already that have been proven useful to users.
So let me just ask you one final question: suppose I do all that
stuff. The question: in your personal opinion, based on the best
information you have, what is the your own subjective estimate of the
probability?


Eventually, with hard work, I estimate the chances at, say, 90%. That
is, eventually, unless the code itself shows flaws, the module *will*
be included. However, initially, when first proposed, the chances are
rather like 10%. I.e. people will initially object. Decision processes
take their time, and valid concerns must be responded to. I personally
think that there is a good response to each concern, but it will take
time to find it. Before that, it will take time to find out what
precisely the concern is.

Regards,
Martin
Jul 18 '05 #78
Paul Rubin wrote:
(And
actually: mxCrypto is the most capable of these packages and might be
the one with the most users, but it's completely unsuitable for the
core because of its size).


mxCrypto is primarily unsuitable for the core because Marc-Andre Lemburg
will never ever contribute it. He is very concerned about including
crypto code with the Python distribution, so he certainly won't
contribute his own.

Regards,
Martin
Jul 18 '05 #79
Paul Rubin <http> wrote:
An AES or DES addition to an existing module that implements just one
call:
ECB(key, data, direction)
would be a huge improvement over what we have now. A more complete
crypto module would have some additional operations, but ECB is the
only one that's really essential.


I would hate to see a module which only implemented ECB. Sure its the
only operation necessary to build the others out of, but its the least
secure mode of any block cipher.

If you don't offer users a choice, then they'll use ECB and just that
along with all its pitfalls, meanwhile thinking that they are secure
because they are using AES/DES...

For those people following along at home (I'm sure everyone who has
contributed to thread knows this already) I tried to find a simple
link to why ECB is bad, this PDF is the best I could come up with, via
Google's Cache.

http://www.google.com/search?q=cache...o/lect04_4.pdf

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick
Jul 18 '05 #80
Paul> I've had this discussion here before, maybe not with you. What I
Paul> really want is zero installations of anything.

Fine. Go build a sumo distribution and track the normal CPython. The
problem isn't all that new. (Take a look at scipy.org for one take on that
theme. Of course Linux distros have been doing their take on this forever.)
If everyone adopted your position that it wasn't Python unless it had
been added to the core, we'd all be reinventing lots of wheels or
tackling much less challenging tasks, if we programmed in Python at
all. Here's an incomplete list of stuff not in the core I have used
happily over the past several years to do my jobs using Python:
Paul> That makes no sense at all. That list is a list of programs
Paul> written in the Python language. They are Python programs, where
Paul> Python is an adjective.

No, many of them are just modules or programming frameworks.
* SpamBayes
Paul> I have the impression this is an application and not a module,

Yes, you're correct.
* Quixote
Paul> Don't know what this is.

Web app framework.
* Docutils
Paul> Should be in the core if it's what I think it is.

Probably will be someday.
* MoinMoin
Paul> Application, should be separate. Also, GPL'd, I think. Can't be
Paul> distributed under PSF license.

Sure.
* Psyco
Paul> I think this isn't ready for prime time yet. Should go into the
Paul> core once it is.

It's getting close for those of us with Intel chips in our boxes.
* PyInline
Paul> Not sure what this is.

A module for inlining C code within a Python module. Also see Weave from
the scipy.org folks. It was inspired by the Perl Inline::C module.
* PyGTK
Paul> wxPython might be a better choice.

Doesn't matter. At work they decreed GTK as the GUI platform long before I
came along (they also use gtkmm for C++ apps). It's still an example of a
broadly useful package available outside the core distribution.
* xmlrpclib before it was in the core
Paul> 1. Did you really need this, instead of some more reasonable rpc
Paul> format?

Yes, for several years I used a homegrown RPC solution behind the Musi-Cal
website that was Python only. Eventually Mojam (a Perl shop) bought
Musi-Cal (a Python shop). I switched to XML-RPC with little effort. At one
point we also had Java talking XML-RPC.

Paul> xdrlib has been in the core forever.

Sure. But it's somewhat lower level than XML-RPC and isn't really an RPC
protocol. It's just a marshalling protocol and is probably not as flexible
as XML-RPC at that.

Paul> 2. Isn't xmlrpclib written in Python?

Yes. The implementation language is just a detail. I also use Fredrik
Lundh's sgmlop library to accelerate XML-RPC and play some other games when
I know I'm talking Python-to-Python (marshal my args, then XML-RPC the
result passing a single argument between the client and server).
* MAL's mx.DateTime before the core datetime module was available
Paul> See, as Python improved, those things went into the core.

Sure, than that's what Martin has been trying to tell you about your AES
proposal. Put it out there, refine it, and get it into the core when it's
mature.
* timeout_socket before sockets supported timeouts
Paul> Could you use sigalarm instead?

I suppose. That's not the point though. I'm not married to the concept as
you seem to be that something has to be in the core distribution to be of
use to me. I'm perfectly happy incorporating solutions other people
provide. I believe you will find I am in the majority in this regard.
Many of those things I could never have written myself, either for
lack of time, lack of skill or both. I'm grateful they were
available when I needed them and feel no qualms about using them even
though they are not distributed with Python proper.
Paul> Sure, it's fine if you have all those modules and you write a
Paul> Python program that uses, say, five of them. External modules
Paul> aren't so bad when the developer and the end user are the same
Paul> person. What happens if you send your Python program to a
Paul> nonprogrammer friend who has just a vanilla Python installation?

I figure out some other packaging solution. In my world most of the
software I write is for my employer, so this is not a problem I face very
often. People use freeze, py2exe, py2app or other packaging solutions to
solve most/all of these problems.

Paul> Now he has to download and install those five modules too. You
Paul> send him the url's where you got the modules a year ago. What are
Paul> the chances that the 5 url's even all still work, much less the
Paul> chance of him being able to install and run all 5 of the modules
Paul> without needing help? What if the versions he downloads (from
Paul> separate developers) have gotten out of sync with each other and
Paul> can't interoperate any more?

This is the well-known "CPAN in Python" problem. People are working on it.
Perhaps you would like to spend some energy helping solve it. If so, join
the catalog-sig.
Notice another interesting feature of several of those items: csv,
xmlrpclib, mx.DateTime, timeout_socket. They were all modules I used
that eventually wound up in the core in some fashion. They didn't go
in the core first, then demonstrate their usefulness. It was the
other way around.


Paul> I'm not sure about timeout_socket and it sounds like it should
Paul> have just been a patch to the socket module, not a new module.

Sure, but a shim between the socket module and Python modules that used it
was a good first approximation to the problem. (I am also a firm believer
in successive approximation to problem solving, especially when I don't know
enough about the problem to know precisely what form the final solution will
take.)

Paul> csv is quite a complex module and took a lot of tweaking and PEP
Paul> editing before standardization. But the need for it was obvious;
Paul> the only good reason it wasn't in the core ages ago was that no
Paul> one had done the work of writing it and shaking it out.

Actually, there were at least two fairly mature implementations of CSV
modules out there before the PEP was a twinkle in anyone's eye. The authors
of those modules got together and wrote the current PEP and module from
scratch based upon their collective experience. I think the effort of
having a couple versions out in the field followed by joint effort to
produce something worthy of inclusion in the core is an excellent
demonstration of what Martin has been saying all along.

Paul> xmlrpclib, not sure. How long was it in separate distribution?

Not all that long. XML-RPC itself hadn't been around very long before
Fredrik wrote xmlrpclib. Both the protocol and xmlrpclib (as well as
similar modules for other languages) caught on pretty quickly.

Skip
Jul 18 '05 #81
"Martin v. Löwis" <ma****@v.loewis.de> writes:
Apparently, people disagree on what precisely the API should be. E.g.
cryptkit has

obj = aes(key)
obj.encrypt(data)
I don't disagree about the API. The cryptkit way is better than ECB
example I gave, but the ECB example shows it's possible to do it in
one call.
I think I would prefer explicit encrypt/decrypt methods over a
direction parameter. Whether or not selection of mode is a separate
parameter, or a separate method, might be debatable
I prefer separate methods too, however if it was done with a direction
flag instead, it wouldn't really cause a problem. As long as the
functionality is there, I can use it.
I would trust my intuition more for a single function than for an
entire API. In this specific proposal, I think I would trust my
intuition and reject the ECB function because of the direction argument.
As an experienced user of a lot of these packages, I can tell you I've
seen it done both ways and I have a slight preference for separate
calls, but it really doesn't matter one way or the other and it's not
worth getting in a debate about it or having a committee design the
API and worry about such trivial issues.

BTW, the main reason to reject the example ECB function is that
creating a key object ("key schedule") from a string can take
significant computation (sort of like compiling a regexp) so the ECB
function for some ciphers would have to cache the object like the
regexp module does. Yuck.

The direction flag question would normally be between:

key = aes.key(key_data)
ciphertext = key(plaintext, "e")

or
key = aes.key(key_data)
ciphertext = key.encrypt(plaintext)

FWIW, another way to do it, also sometimes preferable, is:

key = aes.ecb(key_data, "e") # e for encryption, or "d" for decryption
ciphertext = key(plaintext)

I think the module I proposed did it this last way, but I haven't
looked at it in a while.

The point is that when faced with yet another crypto package, I don't
get in too concerned about which simple API variant it uses to do such
a basic operation. I just care that the operation is available. I
look in the docs to find that package's particular API for that
operation, and I do what the docs say.

I should make it clear that this module is like Python's low-level
"thread" module in that you have to know what you're doing in order to
use it directly without instantly getting in deep trouble. Most
applications would instead use it indirectly through one or more
intermediate layers.
I fully understand what you desire - to include the module "as a
battery". What makes this decision difficult is that you fail to
understand that I don't want included batteries so much that I
would accept empty or leaking batteries.
I do understand that, and the prospect of empty or leaking batteries
is vitally important to considering whether to include a battery
that's included, but for the purposes of an included-battery
discussion, the characteristics of NON-included batteries is not
relevant, given that we know they exist.
http://sourceforge.net/projects/cryptkit/ ...>

I've examined that module, I wouldn't consider it
ideal for the core (besides AES, it has some complicated additional
functions that aren't useful to most people)


Ok, that would be a problem. If this is a simple removal of functions
that you'ld request (which functions?),


OK. First you have to decide whether you want a general crypto
toolkit, or just an AES module. I've been concentrating on just an
AES module (or rather, a generic block cipher module with AES and DES)
since I figure that creates fewer areas of controversy, etc. I think
it's too early to standardize a fancy toolkit. Once there's block
ciphers, we can think about adding more stuff afterwards.

For that module, I'd say remove everything except AES and maybe
SHA256, and ask that DES be added. SHA256 is possibly useful, but
isn't really part of an encryption package; it can be separated out
like the existing sha and md5 modules. Also, it should be brought
into PEP 247 compliance if it's not already.

Rationale: I'd get rid of the entropy module now that os.urandom is
available. Having the OS provide entropy is much better than trying
to do it in user code. I'd get rid of the elliptic curve stuff unless
there's some widely used standard or protocol that needs that
particular implementation. Otherwise, if I want ECC in a Python
program, I'd just do it on characteristic-p curves in pure Python
using Python longs. (Bryan's package uses characteristic-2 curves
which means the arithmetic is all boolean operations, that are more
efficient on binary CPU's, especially small ones. But that means the
module has to be written in C, because doing all those boolean
operations in Python is quite slow. It would be like trying to do
multi-precision arithmetic in Python with Python ints instead of
longs). Once there's a widely accepted standard for ECC like there is
for AES, then I'd want the stdlib to have an implementation of the
standard, but right now there are just a lot of incompatible,
nonstandard approaches running around.

If SHA256 is accepted then SHA512/SHA384 (these are basically the
same) might as well also be. Not many people are using any of these
hash functions right now. Usage will increase over time (they are US
federal standards like AES), and they'll probably be worth adding
eventually. I'm indifferent to whether they're added now.

I think I'd include RC4 under the "toolkit" approach, if it's not
already there. I'd also include a pair of functions like the ones in
p3.py, i.e. an utterly minimal API like:

ciphertext = encrypt_string(key_string, plaintext)
plaintext = decrypt_string(key_string, ciphertext)

that does both encryption and authentication, for key and data strings
of arbitrary size. This would be what people should use instead of
the rotor module. It would be about a 10 line Python function that
calls the block cipher API. The block cipher API itself is intended
for more advanced users.
I may have made exceptions to this rule in the past, e.g. when the
proposal is to simply wrap an existing C API in a Python module
(like shadow passwords). In this case, both the interface and
the implementation are straight-forward, and I expect no surprises.
I'd be happy with an AES module that simply wrapped a C API and that
should be pretty much surprise-free. It would be about like the SHA
module in terms of complexity. What I proposed tries to be a bit more
Pythonic but I can live without that.
For an AES module (or most other modules), I do expect surprises.
Well, the hmac module was added in 2.2 or 2.3, without any fuss. It's
written in Python and is somewhat slow, though. What kind of
development process should it take to replace it in the stdlib with a
C module with the exact same interface?

I think you're imagining a basic AES module to be more complicated
than it really is, because you're not so familiar with this type of
module. Also, possibly because I'm making it sound like a lot of work
to write. But that work is just for the C version, and assumes that
it's me personally writing it. What little experience I've had with
Python's C API has been painful, so I figure on having to spend
considerable time wrestling with it. Someone more adapt with the C
API could probably implement the module with less effort than I'd need.
I have said many times that I am in favour of including an AES
implementation in the Python distribution, e.g. in
Oh, ok. Earlier you said you wanted user feedback before you could
conclude that there was reason to want an AES module at all.
What I cannot promise is to include *your* AES implementation,
not without getting user feedback first.
That's fine. But I think it's reasonable to actually approach some
users and say "this module is being considered for addition to the
core--could you try plugging it into your applications that now use
other external modules, and say whether you think it will fill your
needs".

That's impossible if consideration doesn't even start until testing is
complete. All one could say then is "here's yet another crypto
module, that does less than the toolkit you're already using, could
you please temporarily drop whatever you're doing and update your
programs to switch from your old module that works, to a new module
that MIGHT work?". If the new module is accepted into the core, of
course, it becomes worth retrofitting the existing toolkits to use it.
The whole notion of creating the module from scratch just to include
it in the core strikes me as odd - when there are so many AES
implementations out there already that have been proven useful to users.


We discussed this already. Here are three possible contexts for
designing a crypto module:

1) You're designing it to support some specific application you're
working on. The design will reflect the needs of that application
and might not be so great for a wider range of applications.
2) You're writing a general purpose module for distribution outside
the core (example: mxCrypto). You'll include lots of different
functions, pre-built implementations of a variety of protocols, etc.
You might include bindings that try to be compatible with other
packages, etc. Maybe this can get added to the core someday,
like numarray, but for now, that's a rather big step.
3) You're designing to add basic functionality to the core. Here,
you try to pick a general purpose API not slanted towards a
particular app, and provide just some standard building blocks
that other stuff can be built around. This is more like the
math module, which just does basics: sqrt, sin, cos, etc., with no
attempt at the stuff in a package like numarray. But if there's
a standard like FIPS 80 and if there's good reason to implement
a big subset of it (which there is), then you may as well
implement the standard completely unless there's a good reason
not to (which there isn't; the less important operations are a
few dozen lines of code total).

I think context #3 gets you something better suited for the core and
none of the existing crypto modules were written that way. The same
is in fact true for many of the non-crypto modules, that seem to have
been written in context #1 and would have been better under context #3.

Also, there's the plain fact that none of the authors of the existing
crypto modules have offered to contribute them. So somebody had to
step up and do something.
what is the your own subjective estimate of the probability?

Eventually, with hard work, I estimate the chances at, say, 90%.


Hmm, this is very very interesting. I am highly confident that all
the purely technical questions (i.e. everything about the API and the
code quality, etc.) can converge to a consensus-acceptable solution
without much hassle. I had thought there were insurmountable
obstacles of a nontechnical nature, mainly caused by legal issues, and
that these are beyond any influence that I might have by writing or
releasing anything.
Jul 18 '05 #82
"Martin v. Löwis" <ma****@v.loewis.de> writes:
mxCrypto is primarily unsuitable for the core because Marc-Andre Lemburg
will never ever contribute it. He is very concerned about including
crypto code with the Python distribution, so he certainly won't
contribute his own.


Oh wait, I confused mxcrypto and m2crypto. Sorry. Anyway, the
technical considerations are similar.
Jul 18 '05 #83
Nick Craig-Wood <ni**@craig-wood.com> writes:
I would hate to see a module which only implemented ECB. Sure its the
only operation necessary to build the others out of, but its the least
secure mode of any block cipher.
It's intended as a building block for other modes. Most applications
shouldn't use it directly.
If you don't offer users a choice, then they'll use ECB and just that
along with all its pitfalls, meanwhile thinking that they are secure
because they are using AES/DES...


The documentation has to be written somewhat forcefully to tell users
what not to do. I can help with that. I've had to do that a lot,
supporting crypto packages in projects where the other programmers
haven't used crypto very much.
Jul 18 '05 #84
Skip Montanaro <sk**@pobox.com> writes:
>> * Quixote
Paul> Don't know what this is.

Web app framework.
I think Python should add a web app framework to its core, again since
it otherwise can't seriously begin to compete with PHP. However,
there are lots of approaches so this is an example of where your
suggested process of letting a bunch of different implementations
circulate before choosing something is a good idea.
>> * PyInline
Paul> Not sure what this is.

A module for inlining C code within a Python module. Also see Weave from
the scipy.org folks. It was inspired by the Perl Inline::C module.


Hmm, sounds like it has the same issues as Pyrex. I'm also not sure
why you'd want both PyInline and Pyrex.
>> * PyGTK


Paul> wxPython might be a better choice.

Doesn't matter. At work they decreed GTK as the GUI platform long before I
came along (they also use gtkmm for C++ apps).


Can't wxPython use GTK?
It's still an example of a broadly useful package available outside
the core distribution.
I'd say if access to GTK is widely important functionality, then the
core should provide it in some way (e.g. through wxPython) and that's
enough. If your company wants some different (i.e. nonstandard, if
wxPython becomes the standard) form of access, then it can deal with
the consequences of not following standards.
Paul> 2. Isn't xmlrpclib written in Python?
Yes. The implementation language is just a detail.
I think it's more than a detail. If an external module is written in
Python, I can download it from wherever and include it with my own app
that I send to an end user. I do the work so the end user doesn't
have to. If it's written in C, then the end user has to deal with it.
Paul> See, as Python improved, those things went into the core.

Sure, than that's what Martin has been trying to tell you about your AES
proposal. Put it out there, refine it, and get it into the core when it's
mature.
What kind of refinements are you envisioning? This isn't a web
application framework we're talking about. It's more like the sha
module.
Paul> Could you use sigalarm instead?

I suppose. That's not the point though. I'm not married to the concept as
you seem to be that something has to be in the core distribution to be of
use to me. I'm perfectly happy incorporating solutions other people
provide.
So aren't you happier when the other person provides you with a
solution that installs with one command, instead of a solution that
requires you to download N different modules from who knows where, and
install them separately, all while hoping that they haven't been
tampered with? If I'm trying to provide someone else with a solution,
I'd rather use sigalarm than make the end-user download an extra
module, because I think they'll be happier that way.
Paul> What happens if you send your Python program to a
Paul> nonprogrammer friend who has just a vanilla Python installation?

I figure out some other packaging solution. In my world most of the
software I write is for my employer, so this is not a problem I face very
often. People use freeze, py2exe, py2app or other packaging solutions to
solve most/all of these problems.
Only those people who think that a cross-platform application is one
that works on both XP Home and XP Pro. That does simplify some
things. Life in a cult is often indeed simpler than life in the real
world <wink>.
Actually, there were at least two fairly mature implementations of
CSV modules out there before the PEP was a twinkle in anyone's eye.
The authors of those modules got together and wrote the current PEP
and module from scratch based upon their collective experience.
Yes, CSV is complicated and benefits from that process just like
web app frameworks do. Let's pick another example, the hmac module
that appeared in Python 2.2. It implements the RFC 2104 HMAC algorithm.

Where are the two mature implementations that circulated before the
hmac module was added? Where were the authors pooling their
collective wisdom? Where was the year of user feedback? The answer
is, nothing like that was needed. HMAC is simple enough for a module
author to read RFC 2104 and implement what it says, run some tests,
and declare the module good to go.
I think the effort of having a couple versions out in the field
followed by joint effort to produce something worthy of inclusion in
the core is an excellent demonstration of what Martin has been
saying all along.


Martin is saying the opposite: that he doesn't understand the point of
writing a new module that synthesizes from experiences with old
modules, instead of just using one of the old modules.

I don't think there's a one-size-fits-all answer to any of these
questions. You have to have your hands in the details of a specific
problem, to arrive at the best way to deal with that problem.
Jul 18 '05 #85
Paul Rubin wrote:
Oh, ok. Earlier you said you wanted user feedback before you could
conclude that there was reason to want an AES module at all.
I believe I never said that. I said that I wanted user feedback to
determine whether *this* AES module (where this is either your
from-scratch implementation, or any other specific implementation
contributed) is desirable.
Hmm, this is very very interesting. I am highly confident that all
the purely technical questions (i.e. everything about the API and the
code quality, etc.) can converge to a consensus-acceptable solution
without much hassle. I had thought there were insurmountable
obstacles of a nontechnical nature, mainly caused by legal issues, and
that these are beyond any influence that I might have by writing or
releasing anything.


These obstacles are indeed real. But I believe they are not
unsurmountable. For example, there is the valid complaint that,
in order to export the code from SF, we need to follow U.S.
export laws. 10 years ago, these would have been unsurmountable.
Today, it is still somewhat painful to comply with these laws,
but this is what the PSF is for, which can fill out the forms
necessary to allow exporting this code from the U.S.A.

Regards,
Martin
Jul 18 '05 #86
Skip Montanaro <sk**@pobox.com> wrote:
Fine. Go build a sumo distribution and track the normal CPython.
The problem isn't all that new. (Take a look at scipy.org for one
take on that theme. Of course Linux distros have been doing their
take on this forever.)


If I'm writing code just for fun. I'll be doing on Debian Linux, then
I can do

apt-get install python-crypto

and I'm away.

However if I'm writing code for work, it has to work on windows as
well, which introduces a whole extra barrier to using 3rd party
modules. Something else to compile. Something else to stick in the
installer. Basically a whole heap of extra work.

I think one of the special things about Python is its batteries
included approach, and a crypto library would seem to be an obvious
battery to install since it doesn't (or needn't) depend on any other
library or application.

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick
Jul 18 '05 #87
Paul Rubin <http> wrote:
"Martin v. Löwis" <ma****@v.loewis.de> writes:
Apparently, people disagree on what precisely the API should be. E.g.
cryptkit has

obj = aes(key)
obj.encrypt(data)


I don't disagree about the API. The cryptkit way is better than ECB
example I gave, but the ECB example shows it's possible to do it in
one call.


There is a PEP about this...

API for Block Encryption Algorithms v1.0
http://www.python.org/peps/pep-0272.html

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick
Jul 18 '05 #88
Nick Craig-Wood <ni**@craig-wood.com> writes:
There is a PEP about this...

API for Block Encryption Algorithms v1.0
http://www.python.org/peps/pep-0272.html


Yes, I know about that and have been in contact with its author. He
and I are in agreement (or at least were in agreement some time ago)
that the proposed API of the new module is an improvement, at least
for a generic module. PEP 272 seems to document the interface of
something that had been implemented for some particular application.

Jul 18 '05 #89
"Martin v. Löwis" <ma****@v.loewis.de> writes:
Oh, ok. Earlier you said you wanted user feedback before you could
conclude that there was reason to want an AES module at all.
I believe I never said that. I said that I wanted user feedback to
determine whether *this* AES module (where this is either your
from-scratch implementation, or any other specific implementation
contributed) is desirable.


If that's what you're saying now, I'll accept it and not bother
looking for your other posts that came across much differently.
I had thought there were insurmountable
obstacles of a nontechnical nature, mainly caused by legal issues,..


These obstacles are indeed real. But I believe they are not
unsurmountable. For example, there is the valid complaint that,
in order to export the code from SF, we need to follow U.S.
export laws. 10 years ago, these would have been unsurmountable.


Well, yes, 10 years ago, SF didn't exist <wink>. But there was an ftp
site run by Michael Johnson that had some special server side checks
that made sure the client was in the USA. That was considered good
enough to comply with the export regs, and another guy and I
distributed crypto code (a program that let you use your PC as an
encrypted voice phone) through that site in 1995.

Of course, every time my co-author and I released a new version
through the controlled ftp site, within a day or so the code somehow
managed to show up on another ftp site in Italy with worldwide access.
We (the authors) always managed to be shocked when that happened. But
we had nothing to do with it, so it wasn't our problem.
Today, it is still somewhat painful to comply with these laws,
but this is what the PSF is for, which can fill out the forms
necessary to allow exporting this code from the U.S.A.


Well, complying is painful in the sense of being morally repugnant
(people shouldn't have to notify the government in order to exercise
their free speech rights), but the actual process is pretty easy in
terms of the work required. Python should qualify for the TSU
exception which means you just need to send an email to the BXA,
without needing to fill out any forms. I thought that's what you had
done for the rotor module.
Jul 18 '05 #90

Nick> I think one of the special things about Python is its batteries
Nick> included approach, and a crypto library would seem to be an
Nick> obvious battery to install since it doesn't (or needn't) depend on
Nick> any other library or application.

Obvious for some I suppose (I've never used any). While it might be
convenient to not have to distribute some third party library in addition to
Python, there is a fundamental problem implementing a crypto algorithm from
scratch for inclusion into Python. There is always the problem that the new
code has to be more rigorously tested than typical code and new bugs means a
new distribution of Python, not just a replacement library. A bug in code
that is not security-related generally means something doesn't work and only
rarely means a security hole has been opened on the computer. A bug in
security-related code more often means the latter as well. I'd much rather
trust a widely-disseminated piece of crypto code that is simply wrapped by
Python than one that was written expressly written for Python (and that will
likely not be exercised much outside the Python community).

I realize the sha module is incorporated this way. It has this comment:

* This code for the SHA algorithm was noted as public domain. The original
* headers are pasted below.
*
* Several changes have been made to make it more compatible with the
* Python environment and desired interface.

While I imagine the changes were fairly small, the guys involved are all
very smart, and the code is fairly straightforward (little, if any, memory
allocation going on), there is still the possibility that a bug lurks in
either the incorporated code or in the changes to it. How quickly could the
Python community respond if a bug was found and fixed in the public domain
SHA code? How much harder would it be for people to adapt if they had to
reinstall Python instead of just an external library?

Skip

Jul 18 '05 #91
Skip Montanaro <sk**@pobox.com> writes:
While it might be convenient to not have to distribute some third
party library in addition to Python, there is a fundamental problem
implementing a crypto algorithm from scratch for inclusion into
Python. There is always the problem that the new code has to be
more rigorously tested than typical code
Actually and surprisingly, that's not really true. Crypto algorithms
are pretty straightforward, so if you examine the code and check that
it passes a bunch of test vectors, you can be pretty sure it's
correct. It's much harder to check something like a compiler, which
has a much bigger state space, far more decision points, etc. The
usual bugs in crypto apps (and there are lots of such bugs) are in how
the primitives are used, not in the primitives themselves.
and new bugs means a new distribution of Python, not just a
replacement library.
Why would that be true of a crypto module and not true of, say, the socket
module? If the socket module has a bug that allows a remote takeover of
the application, that's as bad as a crypto bug.
A bug in code that is not security-related generally means something
doesn't work and only rarely means a security hole has been opened
on the computer. A bug in security-related code more often means
the latter as well.
People often don't understand that that almost all code is
security-related. Any code that touches data that came from the
internet is security related. If the math library arctangent function
has a buffer overflow bug triggered by a certain input number, and
someone uses math.arctan in an image viewing program, then maybe a
specially concocted image designed to set off the bug can take over
the user's computer. So even something like math.arctan is security
related.
While I imagine the changes were fairly small, the guys involved are
all very smart, and the code is fairly straightforward (little, if
any, memory allocation going on), there is still the possibility
that a bug lurks in either the incorporated code or in the changes
to it. How quickly could the Python community respond if a bug was
found and fixed in the public domain SHA code? How much harder
would it be for people to adapt if they had to reinstall Python
instead of just an external library?


If they're able to install external libraries, what stops them from
reinstalling a patched sha module?

The hazards of using a crypto module are sort of like the hazards of
using the threading module. Unless you know what you're doing and are
very careful, it's easy to make an error. But the resulting bugs
happen because the module did exactly what you asked for, not because
it did something different from what you asked for.

If you ever write code that uses the Internet, I highly recommend the
book "Security Engineering", by Ross Anderson. It will give you some
idea of what you are up against.
Jul 18 '05 #92
Paul Rubin <http> wrote:
Actually and surprisingly, that's not really true. Crypto algorithms
are pretty straightforward, so if you examine the code and check that
it passes a bunch of test vectors, you can be pretty sure it's
correct.


I was going to write pretty much the same thing.

If a security flaw is found in a block cipher (say) it won't be
because it has a buffer overflow etc, it will be because the algorithm
is flawed. You can't patch up crypto algorithms, you have to throw
them away and start again (you can't have two incompatible versions of
DES for instance).

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick
Jul 18 '05 #93

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

46 posts views Thread by Robin Becker | last post: by
5 posts views Thread by rony steelandt | last post: by
1 post views Thread by Protoman | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by Purva khokhar | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.