468,760 Members | 1,831 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,760 developers. It's quick & easy.

Can you make this faster?

I have a routine that I really need, but it slows down processing
significantly. Can you spot any ineffeciencies in the code?

This code makes a critical function of mine run about 7x slower than
using a prebuilt format string. For maximum flexibility, it would be
best to calculate the format string using this method, so I'd dearly
love to keep it.

def fmtstring(args):
delim = '\0'
fmt = []
fmt.append('<')
for arg in args:
t = type(arg)
if t == types.StringType:
l = len(arg)
fmt.append(str(l) + 's')
elif t == types.IntType:
fmt.append('i')
elif t == types.LongType:
fmt.append('q')
elif t == types.BooleanType:
fmt.append('c')
elif t == types.FloatType:
fmt.append('d')
else:
raise Exception("Can't pack argument of type %s!" % t)
s = ''.join(fmt)
s = s + '\0'
return s
Jul 18 '05 #1
6 1088
In article <88*************************@posting.google.com> ,
kl*******@home.com (Kamilche) wrote:
I have a routine that I really need, but it slows down processing
significantly. Can you spot any ineffeciencies in the code?

This code makes a critical function of mine run about 7x slower than
using a prebuilt format string. For maximum flexibility, it would be
best to calculate the format string using this method, so I'd dearly
love to keep it.

def fmtstring(args):
delim = '\0'
fmt = []
fmt.append('<')
for arg in args:
t = type(arg)
if t == types.StringType:
l = len(arg)
fmt.append(str(l) + 's')
elif t == types.IntType:
fmt.append('i')
elif t == types.LongType:
fmt.append('q')
elif t == types.BooleanType:
fmt.append('c')
elif t == types.FloatType:
fmt.append('d')
else:
raise Exception("Can't pack argument of type %s!" % t)
s = ''.join(fmt)
s = s + '\0'
return s


The first thing is to profile your code and make sure this really is
significant in the overall scheme of things. From your description, it
sounds like you've already done that, so I'll go with that assumption.

What's the most likely value for type(arg)? If 90% of your args are
floats, for example, most of the time you work your way all the way down
the if/else tree before getting to the right spot. Putting the tests in
order of probability of occurance will help you a little.

Doing "from types import *" and then being able to just say "IntType"
instead of "types.IntType" will be a little faster since there will be
one less name lookup.

Another thing you can try is doing dictionary lookups instead of an
if/else tree. Something like:

map = {IntType: 'i',
LongType: 'q',
BooleanType: 'c',
FloatType: 'd'}
[...]
try:
fmt.append (map[t])
except KeyError:
raise Exception ("Can't pack %s" % t)

You still need to special-case StringType, however.

Lastly, is an idea which is really quite gross. If you're desperate for
speed, the majority of your args are strings, and you have no pride, it
might be worth trying, however. Instead of saying:

t = type(arg)
if t == types.StringType:
l = len(arg)

you could do something like:

try:
l = len (arg)
fmt.append(str(l) + 's')
except TypeError:
process all the non-string types

The idea here is to not waste time getting and testing the type, just
assume it's a string and deal with the consequences later if your
assumption was wrong. In real life, this plays out as, "It's easier to
ask forgiveness than permission".

Of the types you mentioned, string is the only one which which has a
length; all the others will raise TypeError when passed to len().
Exception processing is relatively slow, but if it only happens a small
percentage of the time, you might be ahead of the game this way.

Some people would call this elegant, others would call it a gross hack.
I guess it depends on your perspective, and how desperate you are to
make your code run faster :-)

Of course, you could put several of these ideas together. Try the
exception-catching idea for strings first, and then sort out the other 4
types using a dictionary lookup.

My gut feeling is that even with all these tricks, the best you'll be
seeing is a factor of 2 or 3 speedup. I'd be amazed if you could get
anywhere near the 7x you're looking for.
Jul 18 '05 #2
Kamilche wrote:
I have a routine that I really need, but it slows down processing
significantly. Can you spot any ineffeciencies in the code?


I would use a dictionary, and I would pre-allocate the result list:

_code = {types.StringType: 's',
types.IntType: 'i',
types.LongType: 'q',
types.BooleanType: 'c',
types.FloatType: 'd'
}
def fmtstring(args):
fmt = ['\0']*(len(args)+1)
fmt.append('<')
for i, arg in enumerate(args):
t = type(arg)
try:
c = _code[t]
except KeyError:
raise Exception("Can't pack argument of type %s!" % t)
if c == 's':
fmt[i] = str(len(arg))+'s'
else:
fmt[i] = c
return ''.join(fmt)

Regards,
Martin

Jul 18 '05 #3
Kamilche wrote:
I have a routine that I really need, but it slows down processing
significantly. Can you spot any ineffeciencies in the code?


I would use a dictionary, and I would pre-allocate the result list:

_code = {types.StringType: 's',
types.IntType: 'i',
types.LongType: 'q',
types.BooleanType: 'c',
types.FloatType: 'd'
}
def fmtstring(args):
fmt = ['\0']*(len(args)+1)
fmt.append('<')
for i, arg in enumerate(args):
t = type(arg)
try:
c = _code[t]
except KeyError:
raise Exception("Can't pack argument of type %s!" % t)
if c == 's':
fmt[i] = str(len(arg))+'s'
else:
fmt[i] = c
return ''.join(fmt)

Regards,
Martin

Jul 18 '05 #4
On Sun June 27 2004 15:22, Kamilche wrote:
I have a routine that I really need, but it slows down processing
significantly. Can you spot any ineffeciencies in the code?

This code makes a critical function of mine run about 7x slower than
using a prebuilt format string. For maximum flexibility, it would be
best to calculate the format string using this method, so I'd dearly
love to keep it.

def fmtstring(args):
delim = '\0'
fmt = []
fmt.append('<')
for arg in args:
t = type(arg)
if t == types.StringType:
l = len(arg)
fmt.append(str(l) + 's')
elif t == types.IntType:
fmt.append('i')
elif t == types.LongType:
fmt.append('q')
elif t == types.BooleanType:
fmt.append('c')
elif t == types.FloatType:
fmt.append('d')
else:
raise Exception("Can't pack argument of type %s!" % t)
s = ''.join(fmt)
s = s + '\0'
return s


I've tried this:
import types

def fmtstring(args):
delim = '\0'
fmt = []
fmt.append('<')
for arg in args:
t = type(arg)
if t == types.StringType:
l = len(arg)
fmt.append(str(l) + 's')
elif t == types.IntType:
fmt.append('i')
elif t == types.LongType:
fmt.append('q')
elif t == types.BooleanType:
fmt.append('c')
elif t == types.FloatType:
fmt.append('d')
else:
raise Exception("Can't pack argument of type %s!" % t)
s = ''.join(fmt)
s = s + '\0'
return s

typedic = {
types.StringType : 's',
types.IntType: 'i',
types.LongType: 'q',
types.BooleanType: 'c',
types.FloatType:'d'}

def myfmtstring(args):
global typedic
f2 = '<'
t = None
for arg in args:
t = type(arg)
if t == types.StringType:
f2 += str(len(arg))
try:
f2 += typedic[t]
except: #check the exception here!
raise Exception("Can't pack argument of type %s!" % t)
return f2+'\0'

if __name__=='__main__':
import unittest
TIMES = 10000

class TestFmtFuncBase(unittest.TestCase):
def setUp(self):
self.what = ['hola','que','tal',324,454,False]

def runFuncOnce(self):
#shouldnt be here!
assert(False)

def runFuncTimes(self, times=TIMES):
for i in xrange(times):
self.runFuncOnce()
class TestFunction1(TestFmtFuncBase):
def runFuncOnce(self):
return fmtstring(self.what)
def testSpeed(self):
"""Run function a lot of times to check its time"""
self.runFuncTimes()
def testValue(self):
"""Check return value"""
print self.runFuncOnce()

class TestFunction2(TestFmtFuncBase):
def runFuncOnce(self):
return myfmtstring(self.what)
def testSpeed(self):
"""Run function a lot of times to check its time"""
self.runFuncTimes()
def testValue(self):
"""Check return value"""
print self.runFuncOnce()
suiteOriginal = unittest.TestSuite()
suiteOriginal.addTest(unittest.makeSuite(TestFunct ion1))
suiteNew = unittest.TestSuite()
suiteNew.addTest(unittest.makeSuite(TestFunction2) )

unittest.TextTestRunner(verbosity=2).run(suiteOrig inal)
unittest.TextTestRunner(verbosity=2).run(suiteNew)

And seems to be not faster:

----------------------------------------------------------------------
Run function a lot of times to check its time ... ok
Check return value ... <4s3s3siic
ok

----------------------------------------------------------------------
Ran 2 tests in 0.477s

OK
Run function a lot of times to check its time ... ok
Check return value ... <4s3s3siic
ok

----------------------------------------------------------------------
Ran 2 tests in 0.396s

OK
But, anyway, the problem seems to be the:
f2 += str(len(arg))
line.

If you take it off, you'll see that time reduces to half!
Why? How to fix?
Don't know! :-(

Sorry,
dave
--
+ There is no dark side of the moon really. Matter of fact it's all dark.
Jul 18 '05 #5
On Sun June 27 2004 18:51, alejandro david weil wrote:
But, anyway, the problem seems to be the:
f2 += str(len(arg))
line.

If you take it off, you'll see that time reduces to half!
Why? How to fix?
Don't know! :-(


For example, this seems to be "better":


Well, that was not optimized and some debug print remains, this,
uses half time:

typedic = {
types.IntType: 'i',
types.LongType: 'q',
types.BooleanType: 'c',
types.FloatType:'d'}

slen = {}
for i in xrange(256):
slen[i]=str(i)+'s'

def myfmtstring(args):
global typedic, slen
f2 = '<'
t = None
for arg in args:
t = type(arg)
if t == types.StringType:
try:
f2+=slen[len(arg)]
except:
f2 += str(len(arg))+'s'
else:
try:
f2 += typedic[t]
except: #check the exception here!
raise Exception("Can't pack argument of type %s!" % t)
return f2+'\0'

-> Ran 2 tests in 0.253s

Also I tried the except-catch method proposed, in this way, and got ugly results:
(removing StringType from typedic)

def my_slowest_fmtstring(args):
global typedic, slen
f2 = '<'
t = None
for arg in args:
t = type(arg)
try:
f2 += typedic[t]
except: #check the exception here!
if t == types.StringType:
try:
f2+=slen[len(arg)]
except:
f2 += str(len(arg))+'s'
else:
raise Exception("Can't pack argument of type %s!" % t)
return f2+'\0'

-> Ran 2 tests in 0.632s (worst than the first version! that gives: Ran 2 tests in 0.465s)
--
+ There is no dark side of the moon really. Matter of fact it's all dark.
Jul 18 '05 #6
>
But, anyway, the problem seems to be the:
f2 += str(len(arg))
line.

If you take it off, you'll see that time reduces to half!
Why? How to fix?
Don't know! :-(


For example, this seems to be "better":

typedic = {
types.StringType : 's',
types.IntType: 'i',
types.LongType: 'q',
types.BooleanType: 'c',
types.FloatType:'d'}

slen = {}
for i in xrange(256):
slen[i]=str(i)
def myfmtstring(args):
global typedic, slen
f2 = '<'
t = None
for arg in args:
t = type(arg)
if t == types.StringType:
try:
f2+=slen[len(arg)]
except:
print 'aca'
f2 += str(len(arg))
try:
f2 += typedic[t]
except: #check the exception here!
raise Exception("Can't pack argument of type %s!" % t)
return f2+'\0'


--
+ There is no dark side of the moon really. Matter of fact it's all dark.
Jul 18 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by ajikoe | last post: by
19 posts views Thread by pkilambi | last post: by
8 posts views Thread by Scott Emick | last post: by
12 posts views Thread by vunet.us | last post: by
13 posts views Thread by Simply_Red | last post: by
48 posts views Thread by istillshine | last post: by
7 posts views Thread by Steve Bergman | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by Marin | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.