473,387 Members | 1,573 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,387 software developers and data experts.

int() 24 times slower then long() in Python 2.3

When I run the follwing code using Python 2.3:

from time import clock
t1 = clock ()
for i in range (10000): a = int ('bbbbaaaa', 16)
t2 = clock ()
for i in range (10000): a = long ('bbbbaaaa', 16)
t3 = clock ()
print (t2-t1) / (t3-t2)

it prints out: 23.9673206147

That seems to mean that the int-conversion is 24 times slower than the
long conversion.
The reason is that in Python 2.3 the int-conversion generates
warning messages that you *never* see but that consume a *lot* of CPU.

So it may happen that old code performing well under Python 2.2 suddenly
slows down a considerable amount under Python 2.3 without any percievable
change in the output ...

Willem Vree

BTW.
The message that you don't see is: OverflowWarning: string/unicode conversion
and you can make it appear by starting the progam with
Python -Wall script.py

or including the following lines in the source text:
import warnings
warnings.resetwarnings()

I suppose that Python is getting over-engineered by too professional
programmers. The first signs of decay?
Jul 18 '05 #1
10 2192
Willem wrote:
When I run the follwing code using Python 2.3:

from time import clock
t1 = clock ()
for i in range (10000): a = int ('bbbbaaaa', 16)
t2 = clock ()
for i in range (10000): a = long ('bbbbaaaa', 16)
t3 = clock ()
print (t2-t1) / (t3-t2)

it prints out: 23.9673206147

That seems to mean that the int-conversion is 24 times slower than the
long conversion.
The reason is that in Python 2.3 the int-conversion generates
warning messages that you *never* see but that consume a *lot* of CPU. I suppose that Python is getting over-engineered by too professional
programmers. The first signs of decay?


No. I think warning.py needs some _more_ engineering - throw in a dictionary
to speed up repetetive calls or so. In the meantime:

import warnings
def dummy(*args, **kw):
pass
warnings.warn = dummy

Now watch it speed up :)

Peter
Jul 18 '05 #2
On Tuesday 13 July 2004 3:09 pm, Willem wrote:
When I run the follwing code using Python 2.3:

from time import clock
t1 = clock ()
for i in range (10000): a = int ('bbbbaaaa', 16)
t2 = clock ()
for i in range (10000): a = long ('bbbbaaaa', 16)
t3 = clock ()
print (t2-t1) / (t3-t2)

it prints out: 23.9673206147

That seems to mean that the int-conversion is 24 times slower than the
long conversion.
The reason is that in Python 2.3 the int-conversion generates
warning messages that you *never* see but that consume a *lot* of CPU.

So it may happen that old code performing well under Python 2.2 suddenly
slows down a considerable amount under Python 2.3 without any percievable
change in the output ...


I don't think this code did work perfectly well under Python2.2 (see below),
or am I missing something?

Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
for i in range (10000): a = int ('bbbbaaaa', 16) ....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: int() literal too large: bbbbaaaa


--
James Henderson, Logical Progression Ltd
http://www.logicalprogression.net
http://mailmanager.sourceforge.net

Jul 18 '05 #3
Willem wrote:

So it may happen that old code performing well under Python 2.2 suddenly
slows down a considerable amount under Python 2.3 without any percievable
change in the output ...


IMO the speed at which a bunch of invalid conversions are
executed means nothing at all. Could you come up with an example
that show the same symptoms in a meaningful context?

i.

Jul 18 '05 #4
On 13 Jul 2004 14:34:25 -0700, Paul Rubin <"http://phr.cx"@NOSPAM.invalid>
wrote:
"Nick Smallbone" <ni**@nick8325.freeserve.co.uk> writes:
>
> IMO the speed at which a bunch of invalid conversions are
> executed means nothing at all. Could you come up with an example
> that show the same symptoms in a meaningful context?


What do you mean? bbbbaaaa is a hex number.
>>> int('bbbbaaaa', 16)

3149638314L


It's not an int. It has to attempt to convert to int, trap the error
and recover from it, and then convert to a long.

I don't undersand the meaning of int()
If I want an int() (two bytes?) I want two bytes.It should truncate.If the
result is the same with long() a part from the warning
I think int() is unmeaningfull.Transparency is away from warnings.
This python is saying I used the wrong function and there at least two
cases:
I have wrong results.
He is doing my businness.

IMHO I want wrong results.
--
.....lotta dura per la verdura

Jul 18 '05 #5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 14 July 2004 8:32 am, paolo veronelli wrote:
On 13 Jul 2004 14:34:25 -0700, Paul Rubin <"http://phr.cx"@NOSPAM.invalid>

wrote:
"Nick Smallbone" <ni**@nick8325.freeserve.co.uk> writes:
> IMO the speed at which a bunch of invalid conversions are
> executed means nothing at all. Could you come up with an example
> that show the same symptoms in a meaningful context?

What do you mean? bbbbaaaa is a hex number.

>>> int('bbbbaaaa', 16)

3149638314L


It's not an int. It has to attempt to convert to int, trap the error
and recover from it, and then convert to a long.


I don't undersand the meaning of int()
If I want an int() (two bytes?) I want two bytes.It should truncate.If the
result is the same with long() a part from the warning
I think int() is unmeaningfull.Transparency is away from warnings.
This python is saying I used the wrong function and there at least two
cases:
I have wrong results.
He is doing my businness.

IMHO I want wrong results.
--
....lotta dura per la verdura


I think you should look at PEP 237 "Unifying Long Integers and Integers" - it
may even reassure you. As Peter Hansen has already hinted in reply to
another of your messages, Python is in the process of unifying longs and
ints.

It used to be that calling int() on a big number gave an exception - perhaps a
more Pythonic version of your requested giving the wrong result - and now
that it doesn't I suppose there is a sense in which the current distinction
between ints and longs is "unmeaningful" from the user's point of view (it's
not meaningless under the hood of course). According to the PEP the decision
to keep two types was that it:

is far easier to implement, is backwards
compatible at the C API level, and in addition can be implemented
partially as a transitional measure.

Perhaps you thing that ints and longs should not be unified but I won't start
arguing till you come out and say it. :)

James
- --
James Henderson, Logical Progression Ltd
http://www.logicalprogression.net
http://mailmanager.sourceforge.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA9QYUd1yXQ13iWmgRAmmSAKDWOH+vv9RAvPapemqXDS Tax+P1uQCgzbbc
AHRxEquLPpWE32tasFiJOsw=
=taJY
-----END PGP SIGNATURE-----

Jul 18 '05 #6
On Wed, 14 Jul 2004 11:08:16 +0100, James Henderson
<ja***@logicalprogression.net> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 14 July 2004 8:32 am, paolo veronelli wrote:
On 13 Jul 2004 14:34:25 -0700, Paul Rubin
<"http://phr.cx"@NOSPAM.invalid>

wrote:
> "Nick Smallbone" <ni**@nick8325.freeserve.co.uk> writes:
>> > IMO the speed at which a bunch of invalid conversions are
>> > executed means nothing at all. Could you come up with an example
>> > that show the same symptoms in a meaningful context?
>>
>> What do you mean? bbbbaaaa is a hex number.
>>
>> >>> int('bbbbaaaa', 16)
>>
>> 3149638314L
>
> It's not an int. It has to attempt to convert to int, trap the error
> and recover from it, and then convert to a long.
I don't undersand the meaning of int()
If I want an int() (two bytes?) I want two bytes.It should truncate.If
the
result is the same with long() a part from the warning
I think int() is unmeaningfull.Transparency is away from warnings.
This python is saying I used the wrong function and there at least two
cases:
I have wrong results.
He is doing my businness.

IMHO I want wrong results.


I think you should look at PEP 237 "Unifying Long Integers and Integers"
- it
may even reassure you. As Peter Hansen has already hinted in reply to
another of your messages, Python is in the process of unifying longs and
ints.

It used to be that calling int() on a big number gave an exception -
perhaps a
more Pythonic version of your requested giving the wrong result - and now
that it doesn't I suppose there is a sense in which the current
distinction
between ints and longs is "unmeaningful" from the user's point of view
(it's
not meaningless under the hood of course). According to the PEP the
decision
to keep two types was that it:

is far easier to implement, is backwards
compatible at the C API level, and in addition can be implemented
partially as a transitional measure.

Perhaps you thing that ints and longs should not be unified but I won't
start
arguing till you come out and say it. :)

James

I don't want to change any way the base of PEP 237,but this kind of
performance
flaw is a problem and it's possible to be the first of a long list.
It's good to know, always ,why we pay a price.Better an Error then a ghost
warning,which
is itself the cause of the flaw.
Python is fast ,this is why I use it.Anyone has a different goal in using
it.

Regards Paolino


--
.....lotta dura per la verdura

Jul 18 '05 #7
On Wednesday 14 July 2004 12:28 pm, paolo veronelli wrote:

On Wed, 14 Jul 2004 11:08:16 +0100, James Henderson wrote:

I think you should look at PEP 237 "Unifying Long Integers and Integers"
- it
may even reassure you. As Peter Hansen has already hinted in reply to
another of your messages, Python is in the process of unifying longs and
ints.

It used to be that calling int() on a big number gave an exception -
perhaps a
more Pythonic version of your requested giving the wrong result - and now
that it doesn't I suppose there is a sense in which the current
distinction
between ints and longs is "unmeaningful" from the user's point of view
(it's
not meaningless under the hood of course). According to the PEP the
decision
to keep two types was that it:

is far easier to implement, is backwards
compatible at the C API level, and in addition can be implemented
partially as a transitional measure.

Perhaps you thing that ints and longs should not be unified but I won't
start
arguing till you come out and say it. :)

James


I don't want to change any way the base of PEP 237,but this kind of
performance
flaw is a problem and it's possible to be the first of a long list.
It's good to know, always ,why we pay a price.Better an Error then a ghost
warning,which
is itself the cause of the flaw.
Python is fast ,this is why I use it.Anyone has a different goal in using
it.

Regards Paolino


I actually agree it's a shame that even ignored warnings have such a
performance impact but at least these warnings are scheduled to be removed
sometime in Python 2.4 (they're still there in 2.4a1 though). Apparently
they were added for "those concerned about extreme backwards compatibility".

James
--
James Henderson, Logical Progression Ltd
http://www.logicalprogression.net
http://mailmanager.sourceforge.net

Jul 18 '05 #8
James Henderson <ja***@logicalprogression.net> wrote in message news:<ma*************************************@pyth on.org>...
On Wednesday 14 July 2004 12:28 pm, paolo veronelli wrote:

On Wed, 14 Jul 2004 11:08:16 +0100, James Henderson wrote:

I think you should look at PEP 237 "Unifying Long Integers and Integers"
- it
may even reassure you. As Peter Hansen has already hinted in reply to
another of your messages, Python is in the process of unifying longs and
ints.

It used to be that calling int() on a big number gave an exception -
perhaps a
more Pythonic version of your requested giving the wrong result - and now
that it doesn't I suppose there is a sense in which the current
distinction
between ints and longs is "unmeaningful" from the user's point of view
(it's
not meaningless under the hood of course). According to the PEP the
decision
to keep two types was that it:

is far easier to implement, is backwards
compatible at the C API level, and in addition can be implemented
partially as a transitional measure.

Perhaps you thing that ints and longs should not be unified but I won't
start
arguing till you come out and say it. :)

James


I don't want to change any way the base of PEP 237,but this kind of
performance
flaw is a problem and it's possible to be the first of a long list.
It's good to know, always ,why we pay a price.Better an Error then a ghost
warning,which
is itself the cause of the flaw.
Python is fast ,this is why I use it.Anyone has a different goal in using
it.

Regards Paolino


I actually agree it's a shame that even ignored warnings have such a
performance impact but at least these warnings are scheduled to be removed
sometime in Python 2.4 (they're still there in 2.4a1 though). Apparently
they were added for "those concerned about extreme backwards compatibility".

James


That was indeed the reason of my original posting: As a casual
programmer I was disturbed that upgrading to Python 2.3 actually
slowed down my program significantly and that I had to spend so much
time to find the reason: somebody has decided to hide these particular
overflow warnings ...
Why warning and then suppressing it by default?
I understand the problem will disappear in the next version.
I apologize for my remark on "over-engineering". I posted the message
when I was still angry, having spent so much time in finding the "bug"

Willem
Jul 18 '05 #9
On Thursday 15 July 2004 9:58 am, Willem wrote:
That was indeed the reason of my original posting: As a casual
programmer I was disturbed that upgrading to Python 2.3 actually
slowed down my program significantly
As I said before, I don't think the particular code you posted would have
worked at all before 2.3.
and that I had to spend so much
time to find the reason: somebody has decided to hide these particular
overflow warnings ...
Why warning and then suppressing it by default?


The warnings are there because some people may want to know about when the
type conversion is happening, since it's a new feature.

They're suppressed by default because most people won't want to know. :)

J.
--
James Henderson, Logical Progression Ltd
http://www.logicalprogression.net
http://mailmanager.sourceforge.net

Jul 18 '05 #10
James Henderson <ja***@logicalprogression.net> wrote in message news:<ma*************************************@pyth on.org>...
As I said before, I don't think the particular code you posted would have
worked at all before 2.3.


No, indeed. I tried to make a small program that demonstrates the
problem.
But since you are interested, I have made another more accurate
demonstration of my problem.

Willem

--------------------demonstration---------------------------
# we want to read lots of 32 bit CRC values form a text file
# the numbers are hexadecimal ASCII-strings and have to be
# converted to 32-bit signed integers (which happens to be
# the output of zlib.crc32)
import warnings
from time import clock

# be compatible with Python 2.2 for integer overflow
warnings.filterwarnings ('error', category=OverflowWarning)

# my original 8char-hex to 32bit-int conversion in Python 2.2
# it runs much faster for positive numbers than for negative ones
# but is on average faster then an if-based version
a = -158933416 # an example number, make it positive to see quite
# different results
b = '%08x' % a # the hex ascii string
print a, b
t1 = clock()
for i in range (10000):
try: c = int (b, 16)
except: c = - int (0x100000000 - long (b, 16))
t2 = clock()
print t2 - t1, c, type (c)

# straight forward adaptation to Python 2.3: replace try by if.
# first we set warnings to the default
# (this version does not work in Python 2.2)
warnings.filterwarnings ('ignore', category=OverflowWarning)
t1 = clock()
for i in range (10000):
c = int (b, 16)
if c > 0x7fffffff: c = - int (0x100000000 - c)
t2 = clock()
print t2 - t1, c, type (c)

# this adaptation runs 10 times faster then the previous version
# in Python 2.3, but is on average slower than the first version
# in Python 2.2. This is certainly not an obvious adaptation.
# Nobody would expect a long conversion to be, on average, faster
# than an int conversion for 32 bit numbers ...
t1 = clock()
for i in range (10000):
c = long (b, 16)
if c > 0x7fffffff: c = - int (0x100000000 - c)
else: c = int(c)
t2 = clock()
print t2 - t1, c, type (c)
Jul 18 '05 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Peter Kwan | last post by:
Hi, I believe I have discovered a bug in Python 2.3. Could anyone suggest a get around? When I tested my existing Python code with the newly released Python 2.3, I get the following warning: ...
12
by: Matthew Wilson | last post by:
I'm playing around with genetic algorithms and I want to write a function that mutates an integer by iterating across the bits, and about 1 in 10 times, it should switch a zero to a one, or a one...
6
by: Bengt Richter | last post by:
Peculiar boundary cases: >>> 2.0**31-1.0 2147483647.0 >>> int(2147483647.0) 2147483647L >>> int(2147483647L ) 2147483647 >>> >>> -2.0**31
8
by: Gopal | last post by:
Hello: Can anybody suggest a code or modify the following chunk of code for converting a decimal to BCD and viceversa for unsigned long integers. The following code works good for integers but...
16
by: raj | last post by:
Hi, I saw it mentioned that "int" is the fastest data-type for use in C ,that is data storage/retrieval would be the fastest if I use int among the following 4 situations in a 32 bit machine with...
21
by: Andreas Huber | last post by:
Hi there Spending half an hour searching through the archive I haven't found a rationale for the following behavior. using System; // note the missing Flags attribute enum Color {
14
by: Steve McLellan | last post by:
Hi, Sorry to repost, but this is becoming aggravating, and causing me a lot of wasted time. I've got a reasonably large mixed C++ project, and after a number of builds (but not a constant...
87
by: John Rivers | last post by:
Hello everybody, I just wondered if anybody else has noticed this? It takes around 6 seconds to start debugging a very simple ASPX page with VS.NET whereas VB6 takes under 0.5 seconds, even...
35
by: pinkfloydhomer | last post by:
How do I check if a string contains (can be converted to) an int? I want to do one thing if I am parsing and integer, and another if not. /David
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.